[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:30 dist-test-slave-pxr1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:30 dist-test-slave-pxr1 krb5kdc[399](info): set up 2 sockets
Sep 28 22:44:30 dist-test-slave-pxr1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 22:44:31.141052 392 mini_kdc.cc:121] Time spent starting KDC: real 2.223s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 22:44:31.314555 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.173s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 22:44:31.527407 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.213s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 22:44:31.761029 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.234s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/test-user.keytab.
W20260928 22:44:32.315191 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.554s user 0.001s sys 0.000s
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:32.355783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:32.434443 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:32.434795 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:32.434847 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:32.436548 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:32.436612 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:32.436632 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:32.436650 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:32.436666 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:32.439054 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:32.439373 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:32.439687 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:32.445823 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.445990 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.446048 412 server_base.cc:1034] running on GCE node
W20260928 22:44:32.446215 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.446472 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:32.446841 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:32.447995 412 hybrid_clock.cc:648] HybridClock initialized: now 1790635472447968 us; error 40 us; skew 500 ppm
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:32.451035 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:32.451501 412 webserver.cc:460] Webserver started at http://127.0.98.62:45677/ using document root <none> and password file <none>
I20260928 22:44:32.451812 412 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:32.451917 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:32.452073 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:32.453536 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "6176fa7f587d41e797ae1591f10efe9e"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "7fbb862063a0fd8d9e0a62bdfcdf3e2e"
server_key_iv: "e5c9b7634af9f0b20f9affef3e79fbd6"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.453974 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "6176fa7f587d41e797ae1591f10efe9e"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "7fbb862063a0fd8d9e0a62bdfcdf3e2e"
server_key_iv: "e5c9b7634af9f0b20f9affef3e79fbd6"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.455665 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:32.456550 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.456761 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:32.456856 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "6176fa7f587d41e797ae1591f10efe9e"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "7fbb862063a0fd8d9e0a62bdfcdf3e2e"
server_key_iv: "e5c9b7634af9f0b20f9affef3e79fbd6"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.456929 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:32.474416 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:32.477056 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:32.477224 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:32.481725 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40237
I20260928 22:44:32.481767 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40237 every 8 connection(s)
I20260928 22:44:32.482090 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:32.482985 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:32.485409 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: Bootstrap starting.
I20260928 22:44:32.485978 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:32.486229 480 log.cc:824] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:32.486858 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: No bootstrap required, opened a new log
I20260928 22:44:32.488384 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } }
I20260928 22:44:32.488521 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:32.488552 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6176fa7f587d41e797ae1591f10efe9e, State: Initialized, Role: FOLLOWER
I20260928 22:44:32.488698 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } }
I20260928 22:44:32.488797 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:32.488821 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:32.488862 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:32.489358 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } }
I20260928 22:44:32.489447 480 leader_election.cc:302] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6176fa7f587d41e797ae1591f10efe9e; no voters:
I20260928 22:44:32.489591 480 leader_election.cc:288] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:32.489617 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:32.489737 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [term 1 LEADER]: Becoming Leader. State: Replica: 6176fa7f587d41e797ae1591f10efe9e, State: Running, Role: LEADER
I20260928 22:44:32.489861 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:32.489853 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } }
I20260928 22:44:32.490407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 412
I20260928 22:44:32.490513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:32.491196 392 external_mini_cluster.cc:1428] Setting key 5591ac0a498ad7a7b4204897d6f51404
W20260928 22:44:32.491525 498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 22:44:32.491993 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6176fa7f587d41e797ae1591f10efe9e. Latest consensus state: current_term: 1 leader_uuid: "6176fa7f587d41e797ae1591f10efe9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } } }
I20260928 22:44:32.492460 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:32.492148 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6176fa7f587d41e797ae1591f10efe9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6176fa7f587d41e797ae1591f10efe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40237 } } }
I20260928 22:44:32.492624 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e [sys.catalog]: This master's current role is: LEADER
W20260928 22:44:32.492668 498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:44:32.492753 502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:32.492942 502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:32.494264 502 catalog_manager.cc:1347] Generated new cluster ID: c3629decd41d4410b77e48d91e2bd98f
I20260928 22:44:32.494313 502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:32.506407 502 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:32.507077 502 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:32.510231 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.492261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.98.62:40237)
0928 22:44:32.492418 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:32.492428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:32.495246 (+ 2818us) server_negotiation.cc:407] Connection header received
0928 22:44:32.495376 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:32.495382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:32.495466 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:32.495606 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:32.497559 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.499564 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.501025 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.501672 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.501780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.506097 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:32.506101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:32.506101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:32.506114 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:32.508053 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.508586 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.508589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.508590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.508691 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.508991 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.508991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.508992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.509259 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:32.509413 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:32.509694 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 22:44:32.509834 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 22:44:32.515498 502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6176fa7f587d41e797ae1591f10efe9e: Generated new TSK 0
I20260928 22:44:32.515679 502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:32.545862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:32.625308 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:32.625530 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:32.625563 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:32.626716 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:32.626758 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:32.626794 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:32.628262 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:32.628515 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:32.628762 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:32.633479 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.633600 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.633728 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.633603 506 server_base.cc:1034] running on GCE node
I20260928 22:44:32.634016 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:32.634275 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:32.635421 506 hybrid_clock.cc:648] HybridClock initialized: now 1790635472635408 us; error 30 us; skew 500 ppm
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:32.637933 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:32.638329 506 webserver.cc:460] Webserver started at http://127.0.98.1:36689/ using document root <none> and password file <none>
I20260928 22:44:32.638521 506 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:32.638559 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:32.638651 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:32.639679 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "997b31e672904556893eecac2799b6be"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "6f93b984aed85c550066b6146279caac"
server_key_iv: "f930597598cdf9127c43b4d3a6041880"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.639991 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "997b31e672904556893eecac2799b6be"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "6f93b984aed85c550066b6146279caac"
server_key_iv: "f930597598cdf9127c43b4d3a6041880"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.641374 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:32.642132 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.642313 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:32.642396 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "997b31e672904556893eecac2799b6be"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "6f93b984aed85c550066b6146279caac"
server_key_iv: "f930597598cdf9127c43b4d3a6041880"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.642463 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:32.663220 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:32.665381 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:32.665503 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:32.665688 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:32.665946 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:32.665974 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.665995 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:32.666013 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.671747 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41609
I20260928 22:44:32.671824 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41609 every 8 connection(s)
I20260928 22:44:32.672115 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:32.680919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 506
I20260928 22:44:32.680950 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.672854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45517 (local address 127.0.98.62:40237)
0928 22:44:32.672915 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:32.672917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:32.673111 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:44:32.673624 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:32.673625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:32.673637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:32.673685 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:32.674634 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.675133 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.676114 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.676316 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.676371 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.678550 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:32.678553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:32.678553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:32.678561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:32.679671 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.680044 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.680045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.680046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.680099 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.680312 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.680312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.680313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.680426 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:32.680480 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:32.680766 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 22:44:32.680860 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:32.681008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:32.681115 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.672820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40237 (local address 127.0.98.1:45517)
0928 22:44:32.672989 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:32.673027 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:32.673527 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:32.673774 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:32.673777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:32.674338 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:32.674533 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.674538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.675213 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.675215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.676027 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.676029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.676383 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.676384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.676508 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.677058 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:32.677067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:32.678449 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:32.679756 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.679763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.679765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.679977 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.680153 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.680153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.680154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.680257 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.680479 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:32.680486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:32.680619 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 22:44:32.680709 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:44:32.680892 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 22:44:32.681165 392 external_mini_cluster.cc:1428] Setting key 45b993ae84f2767f2a4c9c3e4853e086
I20260928 22:44:32.681784 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40237
I20260928 22:44:32.681891 635 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:32.682065 635 heartbeater.cc:511] Master 127.0.98.62:40237 requested a full tablet report, sending...
I20260928 22:44:32.682552 444 ts_manager.cc:194] Registered new tserver with Master: 997b31e672904556893eecac2799b6be (127.0.98.1:41609)
I20260928 22:44:32.683382 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45517
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:32.705379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:32.784595 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:32.784847 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:32.784884 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:32.786080 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:32.786127 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:32.786164 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:32.787653 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:32.787899 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:32.788110 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:32.793390 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.793502 640 server_base.cc:1034] running on GCE node
W20260928 22:44:32.793417 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.793394 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.793779 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:32.794026 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:32.795159 640 hybrid_clock.cc:648] HybridClock initialized: now 1790635472795149 us; error 31 us; skew 500 ppm
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:32.797479 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:32.797904 640 webserver.cc:460] Webserver started at http://127.0.98.2:38239/ using document root <none> and password file <none>
I20260928 22:44:32.798115 640 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:32.798164 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:32.798285 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:32.799307 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5acc5e58ae0484c9368356b9263eebb"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "07d54c2be19e535e2a47ff2c59896bed"
server_key_iv: "4017f33886cbda091c3c36f9f4152231"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.799638 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5acc5e58ae0484c9368356b9263eebb"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "07d54c2be19e535e2a47ff2c59896bed"
server_key_iv: "4017f33886cbda091c3c36f9f4152231"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.800884 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:32.801573 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.801702 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.801781 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "d5acc5e58ae0484c9368356b9263eebb"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "07d54c2be19e535e2a47ff2c59896bed"
server_key_iv: "4017f33886cbda091c3c36f9f4152231"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.801836 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:32.817584 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:32.819392 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:32.819527 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:32.819738 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:32.820034 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:32.820068 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.820098 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:32.820122 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.826015 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41363
I20260928 22:44:32.826068 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41363 every 8 connection(s)
I20260928 22:44:32.826370 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:32.829843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 640
I20260928 22:44:32.829936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:32.830083 392 external_mini_cluster.cc:1428] Setting key 2dff6601cbb47974006dd50673a341c7
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:32.835317 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.827024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36657 (local address 127.0.98.62:40237)
0928 22:44:32.827085 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:32.827086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:32.827286 (+ 200us) server_negotiation.cc:407] Connection header received
0928 22:44:32.827799 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:32.827800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:32.827812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:32.827864 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:32.828799 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.829292 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.830376 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.830568 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.830623 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.832796 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:32.832798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:32.832798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:32.832805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:32.833996 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.834457 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.834458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.834458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.834503 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.834731 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.834731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.834732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.834829 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:32.834873 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:32.835189 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 22:44:32.835252 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:32.835585 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.826995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40237 (local address 127.0.98.2:36657)
0928 22:44:32.827160 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:32.827196 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:32.827702 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:32.827936 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:32.827940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:32.828506 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:32.828699 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.828704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.829373 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.829374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.830310 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.830313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.830636 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.830637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.830768 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.831256 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:32.831265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:32.832690 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:32.834068 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.834076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.834078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.834389 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.834543 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.834543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.834544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.834675 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.834882 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:32.834888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:32.835057 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 22:44:32.835174 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 22:44:32.835342 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260928 22:44:32.836053 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40237
I20260928 22:44:32.836161 769 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:32.836349 769 heartbeater.cc:511] Master 127.0.98.62:40237 requested a full tablet report, sending...
I20260928 22:44:32.836747 444 ts_manager.cc:194] Registered new tserver with Master: d5acc5e58ae0484c9368356b9263eebb (127.0.98.2:41363)
I20260928 22:44:32.837116 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:32.854969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:32.936468 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:32.936685 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:32.936709 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:32.938015 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:32.938064 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:32.938091 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:32.939611 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:32.939823 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:32.940019 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:32.944747 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.944844 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:32.944918 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:32.944928 774 server_base.cc:1034] running on GCE node
I20260928 22:44:32.945194 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:32.945447 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:32.946605 774 hybrid_clock.cc:648] HybridClock initialized: now 1790635472946584 us; error 36 us; skew 500 ppm
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:32.949465 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:32.949916 774 webserver.cc:460] Webserver started at http://127.0.98.3:45601/ using document root <none> and password file <none>
I20260928 22:44:32.950129 774 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:32.950176 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:32.950300 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:32.951337 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "8188596523f7477d9d21cfed1decb206"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "9958f99276ebc65ed59c581c78076171"
server_key_iv: "8507550bbe5f6072deb5b2a0be8906e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.951670 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8188596523f7477d9d21cfed1decb206"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "9958f99276ebc65ed59c581c78076171"
server_key_iv: "8507550bbe5f6072deb5b2a0be8906e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.952939 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:32.953578 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.953719 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:32.953797 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "8188596523f7477d9d21cfed1decb206"
format_stamp: "Formatted at 2026-09-28 22:44:32 on dist-test-slave-pxr1"
server_key: "9958f99276ebc65ed59c581c78076171"
server_key_iv: "8507550bbe5f6072deb5b2a0be8906e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:32.953856 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:32.962725 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:32.964803 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:32.964938 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:32.965137 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:32.965409 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:32.965441 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.965469 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:32.965493 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:32.971930 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39411
I20260928 22:44:32.971966 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39411 every 8 connection(s)
I20260928 22:44:32.972268 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:32.979583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 774
I20260928 22:44:32.979686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:32.979858 392 external_mini_cluster.cc:1428] Setting key b372d3b85cc1ec74ffb67236522d4b5b
I20260928 22:44:32.980829 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.972977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58233 (local address 127.0.98.62:40237)
0928 22:44:32.973027 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:32.973029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:32.973214 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:44:32.973660 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:32.973660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:32.973671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:32.973721 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:32.974589 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.975072 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.975983 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.976160 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.976202 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.978218 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:32.978219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:32.978220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:32.978226 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:32.979279 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.979761 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.979762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.979763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.979819 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:32.980084 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:32.980085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:32.980085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:32.980225 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:32.980284 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:32.980638 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 22:44:32.980742 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:44:32.981050 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:32.972968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40237 (local address 127.0.98.3:58233)
0928 22:44:32.973108 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:32.973143 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:32.973588 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:32.973776 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:32.973779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:32.974325 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:32.974518 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.974523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.975134 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.975135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.975923 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:32.975925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:32.976212 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:32.976213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:32.976304 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:32.976813 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:32.976821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:32.978117 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:32.979350 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.979356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.979359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.979650 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.979885 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:32.979886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:32.979887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:32.980023 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:32.980312 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:32.980319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:32.980491 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 22:44:32.980609 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 22:44:32.980810 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 22:44:32.981469 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40237
I20260928 22:44:32.981544 903 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:32.981709 903 heartbeater.cc:511] Master 127.0.98.62:40237 requested a full tablet report, sending...
I20260928 22:44:32.982084 444 ts_manager.cc:194] Registered new tserver with Master: 8188596523f7477d9d21cfed1decb206 (127.0.98.3:39411)
I20260928 22:44:32.982450 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58233
I20260928 22:44:32.991964 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:44:32 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:33.013408 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:33.006375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.98.1:41609)
0928 22:44:33.006549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:33.006554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:33.006565 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:33.006590 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:33.006592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:33.006606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:33.006674 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:33.007159 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:33.007879 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:33.008572 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:33.008939 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:33.008992 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:33.011103 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:33.011112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:33.011113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:33.011119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:33.012398 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:33.012706 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:33.012709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:33.012710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:33.012842 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:33.012991 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:33.012991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:33.012991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:33.013117 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:33.013226 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:33.013261 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 22:44:33.013322 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260928 22:44:33.013759 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38800
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:33.034632 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:33.028295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.0.98.1:41609)
0928 22:44:33.028357 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:33.028360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:33.028409 (+ 49us) server_negotiation.cc:407] Connection header received
0928 22:44:33.028498 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:33.028499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:33.028512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:33.028554 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:33.029155 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:33.029648 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:33.030349 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:33.030556 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:33.030611 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:33.032581 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:33.032583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:33.032583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:33.032593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:33.033674 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:33.033967 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:33.033968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:33.033968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:33.034023 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:33.034183 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:33.034184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:33.034184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:33.034280 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:33.034385 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:33.034459 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 22:44:33.034541 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:44:33.035143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 506
I20260928 22:44:33.039938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 640
I20260928 22:44:33.044824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 774
I20260928 22:44:33.048467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 412
2026-09-28T22:44:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4155 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[917](info): set up 2 sockets
Sep 28 22:44:33 dist-test-slave-pxr1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 22:44:33.983950 392 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:34.091339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:34.170971 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:34.171240 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:34.171273 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:34.172438 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:34.172478 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:34.172498 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:34.172513 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:34.172529 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:34.174067 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:34.174324 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:34.174527 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:34.178987 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:34.179082 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:34.179018 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.179177 930 server_base.cc:1034] running on GCE node
I20260928 22:44:34.179438 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:34.179693 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:34.180825 930 hybrid_clock.cc:648] HybridClock initialized: now 1790635474180805 us; error 36 us; skew 500 ppm
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:34.183142 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:34.183522 930 webserver.cc:460] Webserver started at http://127.0.98.62:34205/ using document root <none> and password file <none>
I20260928 22:44:34.183703 930 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:34.183741 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:34.183840 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:34.184896 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "dfd5117823ac4ab4a2c770a257aa19b3"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "c1fab776b6014dca1f7ef8ce57a8b8bf"
server_key_iv: "b8067b21e827ae50b1ae28f532441151"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.185207 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfd5117823ac4ab4a2c770a257aa19b3"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "c1fab776b6014dca1f7ef8ce57a8b8bf"
server_key_iv: "b8067b21e827ae50b1ae28f532441151"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.186403 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:34.187062 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.187223 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:34.187301 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "dfd5117823ac4ab4a2c770a257aa19b3"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "c1fab776b6014dca1f7ef8ce57a8b8bf"
server_key_iv: "b8067b21e827ae50b1ae28f532441151"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.187363 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:34.205444 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:34.207180 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:34.207304 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:34.210851 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42007
I20260928 22:44:34.210887 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42007 every 8 connection(s)
I20260928 22:44:34.211196 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:34.211977 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:34.214166 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3: Bootstrap starting.
I20260928 22:44:34.214710 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:34.214972 998 log.cc:824] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:34.215483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 930
I20260928 22:44:34.215550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:34.215605 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3: No bootstrap required, opened a new log
I20260928 22:44:34.215684 392 external_mini_cluster.cc:1428] Setting key ebd09d5c9c2b67e03554d2e47d829295
I20260928 22:44:34.217350 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260928 22:44:34.217484 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:34.217516 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd5117823ac4ab4a2c770a257aa19b3, State: Initialized, Role: FOLLOWER
I20260928 22:44:34.217638 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260928 22:44:34.217708 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:34.217741 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:34.217788 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:34.218628 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260928 22:44:34.218722 998 leader_election.cc:302] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfd5117823ac4ab4a2c770a257aa19b3; no voters:
I20260928 22:44:34.218880 998 leader_election.cc:288] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:34.218910 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:34.219007 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [term 1 LEADER]: Becoming Leader. State: Replica: dfd5117823ac4ab4a2c770a257aa19b3, State: Running, Role: LEADER
I20260928 22:44:34.219084 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260928 22:44:34.219117 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:34.219465 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfd5117823ac4ab4a2c770a257aa19b3. Latest consensus state: current_term: 1 leader_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } } }
I20260928 22:44:34.219472 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd5117823ac4ab4a2c770a257aa19b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } } }
I20260928 22:44:34.219552 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:34.219581 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:34.219751 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:34.220264 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:34.221794 1009 catalog_manager.cc:1347] Generated new cluster ID: 68bb562a92b54a558cf81a31be3c410a
I20260928 22:44:34.221856 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:34.225965 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.216171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.62:42007)
0928 22:44:34.216356 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.216363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.216388 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:44:34.217312 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.217319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.217366 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.217486 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.218141 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.219148 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.220161 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.220760 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.220865 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.223069 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.223073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.223073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.223084 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.224551 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.224925 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.224928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.224928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.225019 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.225189 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.225189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.225190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.225413 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.225555 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.225645 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.225784 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260928 22:44:34.234030 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:34.234522 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:34.238083 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfd5117823ac4ab4a2c770a257aa19b3: Generated new TSK 0
I20260928 22:44:34.238238 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:34.262907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:34.345659 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:34.345873 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:34.345916 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:34.347110 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:34.347170 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:34.347218 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:34.348767 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:34.349017 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:34.349248 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:34.354056 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:34.354063 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.354171 1024 server_base.cc:1034] running on GCE node
W20260928 22:44:34.354094 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.354513 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:34.354756 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:34.355891 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790635474355885 us; error 30 us; skew 500 ppm
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:34.358358 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:34.358747 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43765/ using document root <none> and password file <none>
I20260928 22:44:34.358939 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:34.358978 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:34.359068 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:34.360061 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c2fe17e071247669154f9d56a672f3e"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "ede3463de5a1fe1ba325f615a0517a98"
server_key_iv: "86a3174da30016869c5ce7ea28fdb256"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.360354 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c2fe17e071247669154f9d56a672f3e"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "ede3463de5a1fe1ba325f615a0517a98"
server_key_iv: "86a3174da30016869c5ce7ea28fdb256"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.361586 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:34.362215 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.362365 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:34.362442 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "4c2fe17e071247669154f9d56a672f3e"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "ede3463de5a1fe1ba325f615a0517a98"
server_key_iv: "86a3174da30016869c5ce7ea28fdb256"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.362509 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:34.381575 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:34.383316 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:34.383446 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:34.383651 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:34.383931 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:34.383962 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.383983 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:34.384006 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.389310 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35267
I20260928 22:44:34.389358 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35267 every 8 connection(s)
I20260928 22:44:34.389642 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:34.397802 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.390372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34567 (local address 127.0.98.62:42007)
0928 22:44:34.390428 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.390430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.390615 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:44:34.391119 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.391120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.391133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.391183 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.392057 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.392547 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.393542 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.393733 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.393783 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.395647 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.395648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.395649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.395656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.396782 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.397107 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.397108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.397108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.397153 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.397342 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.397343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.397343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.397435 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.397478 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.397678 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.397739 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:44:34.397919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1024
I20260928 22:44:34.398008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:34.398013 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.390353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.1:34567)
0928 22:44:34.390504 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.390539 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.391037 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.391255 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.391259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.391795 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:34.391988 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.391992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.392617 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.392618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.393478 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.393481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.393788 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.393789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.393883 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.394306 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:34.394314 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:34.395572 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:34.396851 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.396855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.396857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.397056 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.397207 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.397208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.397208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.397301 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.397475 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:34.397477 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:34.397584 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.397657 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.397798 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260928 22:44:34.398171 392 external_mini_cluster.cc:1428] Setting key c7c96c17cf8bd431890fdc3f8a7b50b2
I20260928 22:44:34.398442 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260928 22:44:34.398540 1153 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:34.398756 1153 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
I20260928 22:44:34.399232 962 ts_manager.cc:194] Registered new tserver with Master: 4c2fe17e071247669154f9d56a672f3e (127.0.98.1:35267)
I20260928 22:44:34.399941 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34567
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:34.421366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:34.499073 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:34.499286 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:34.499318 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:34.500511 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:34.500554 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:34.500589 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:34.502044 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:34.502283 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:34.502494 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:34.507275 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:34.507292 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.507336 1158 server_base.cc:1034] running on GCE node
W20260928 22:44:34.507308 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.507642 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:34.507872 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:34.509006 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790635474508982 us; error 41 us; skew 500 ppm
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:34.511391 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:34.511808 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34059/ using document root <none> and password file <none>
I20260928 22:44:34.512004 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:34.512043 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:34.512135 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:34.513180 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "b5421895e62e452fb91d946e032caeff"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "14b0ad81a5d9007a2925141db9383dca"
server_key_iv: "865c4cabeb25bc743e6f8b063df88934"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.513494 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b5421895e62e452fb91d946e032caeff"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "14b0ad81a5d9007a2925141db9383dca"
server_key_iv: "865c4cabeb25bc743e6f8b063df88934"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.514639 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:34.515265 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.515421 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:34.515491 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "b5421895e62e452fb91d946e032caeff"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "14b0ad81a5d9007a2925141db9383dca"
server_key_iv: "865c4cabeb25bc743e6f8b063df88934"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.515543 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:34.530702 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:34.532397 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:34.532503 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:34.532682 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:34.532994 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:34.533026 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.533046 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:34.533063 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.538381 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45709
I20260928 22:44:34.538436 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45709 every 8 connection(s)
I20260928 22:44:34.538718 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:34.545600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1158
I20260928 22:44:34.545680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:34.545827 392 external_mini_cluster.cc:1428] Setting key 3e9a87ab8ff32a50030f3e37931217e0
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:34.552245 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.539370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55659 (local address 127.0.98.62:42007)
0928 22:44:34.539446 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.539448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.539615 (+ 167us) server_negotiation.cc:407] Connection header received
0928 22:44:34.540061 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.540062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.540071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.540119 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.541043 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.541546 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.542468 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.542644 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.542686 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.549434 (+ 6748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.549437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.549437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.549445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.550961 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.551349 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.551351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.551351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.551418 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.551621 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.551622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.551622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.551763 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.551836 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.552067 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.552148 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:44:34.552337 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.539354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.2:55659)
0928 22:44:34.539498 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.539535 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.539987 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.540176 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.540180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.540767 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:34.540974 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.540979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.541610 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.541610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.542410 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.542412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.542695 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.542696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.542791 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.547888 (+ 5097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:34.547899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:34.549329 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:34.551045 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.551052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.551054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.551279 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.551476 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.551476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.551477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.551571 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.551831 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:34.551833 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:34.551955 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.552030 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.552150 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 22:44:34.552785 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260928 22:44:34.552883 1287 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:34.553058 1287 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 22:44:34.553433 962 ts_manager.cc:194] Registered new tserver with Master: b5421895e62e452fb91d946e032caeff (127.0.98.2:45709)
I20260928 22:44:34.553802 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:34.569023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:34.647010 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:34.647270 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:34.647315 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:34.648515 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:34.648557 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:34.648592 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:34.650058 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:34.650303 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:34.650518 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:34.655303 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.655493 1292 server_base.cc:1034] running on GCE node
W20260928 22:44:34.655308 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:34.655324 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:34.655746 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:34.655985 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:34.657125 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790635474657110 us; error 30 us; skew 500 ppm
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:34.659449 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:34.659909 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42277/ using document root <none> and password file <none>
I20260928 22:44:34.660126 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:34.660176 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:34.660312 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:34.661404 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "81bfaa0bd1f741abb2e93549497484e8"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "5f6f2d923f85811db4ce6322e9fafb04"
server_key_iv: "6e3b9dd6aa56551d538e4494aafa7dde"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.661751 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81bfaa0bd1f741abb2e93549497484e8"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "5f6f2d923f85811db4ce6322e9fafb04"
server_key_iv: "6e3b9dd6aa56551d538e4494aafa7dde"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.663028 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:34.663697 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.663848 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:34.663929 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "81bfaa0bd1f741abb2e93549497484e8"
format_stamp: "Formatted at 2026-09-28 22:44:34 on dist-test-slave-pxr1"
server_key: "5f6f2d923f85811db4ce6322e9fafb04"
server_key_iv: "6e3b9dd6aa56551d538e4494aafa7dde"
server_key_version: "encryptionkey@0"
I20260928 22:44:34.663998 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:34.677464 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:34.679152 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:34.679282 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:34.679483 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:34.679773 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:34.679813 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.679842 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:34.679867 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:34.685220 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42663
I20260928 22:44:34.685272 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42663 every 8 connection(s)
I20260928 22:44:34.685550 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:34.693281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1292
I20260928 22:44:34.693364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:34.693544 392 external_mini_cluster.cc:1428] Setting key 754507b815afab379ee44908c3d0d12e
I20260928 22:44:34.694386 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.686153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57395 (local address 127.0.98.62:42007)
0928 22:44:34.686195 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.686197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.686403 (+ 206us) server_negotiation.cc:407] Connection header received
0928 22:44:34.686866 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.686867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.686875 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.686915 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.687819 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.688307 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.689337 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.689522 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.689574 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.691723 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.691725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.691725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.691732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.693041 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.693396 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.693397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.693398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.693479 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.693711 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.693712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.693712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.693847 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.693913 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.694218 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.694292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:34.694597 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.686130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.3:57395)
0928 22:44:34.686286 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.686326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.686777 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.686985 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.686988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.687546 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:34.687744 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.687748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.688369 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.688369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.689255 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.689258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.689582 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.689582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.689688 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.690215 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:34.690224 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:34.691638 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:34.693103 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.693107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.693108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.693316 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.693536 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:34.693537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:34.693537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:34.693645 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:34.693914 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:34.693920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:34.694084 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.694198 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.694381 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260928 22:44:34.695076 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260928 22:44:34.695199 1421 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:34.695372 1421 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
I20260928 22:44:34.695680 962 ts_manager.cc:194] Registered new tserver with Master: 81bfaa0bd1f741abb2e93549497484e8 (127.0.98.3:42663)
I20260928 22:44:34.696029 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57395
I20260928 22:44:34.705113 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:44:34.711882 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.706630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.98.62:42007)
0928 22:44:34.706702 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.706704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.706762 (+ 58us) server_negotiation.cc:407] Connection header received
0928 22:44:34.706808 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.706808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.706816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.706860 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.707399 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.707877 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.708536 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.708749 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.708806 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.709905 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.709906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.709907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.709912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.711026 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.711314 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.711315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.711315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.711359 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.711473 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.711473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.711474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.711598 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.711665 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.711749 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.711806 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:44:34.714063 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:34.715557 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:34.721873 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42663 (local address 127.0.0.1:35168)
0928 22:44:34.717744 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.717790 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.717854 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.718263 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.718266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.718268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:34.718412 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.718414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719806 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.719807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.720639 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.720641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721551 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.721629 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721644 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.721713 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.721768 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260928 22:44:34.721868 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.98.3:42663)
0928 22:44:34.718052 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.718056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.718067 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:34.718105 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.718107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.718123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.718190 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:34.718488 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719738 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.720747 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721468 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721543 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721607 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.721737 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.721792 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
I20260928 22:44:34.722031 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.1:35267)
0928 22:44:34.718208 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.718212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.718220 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:34.718253 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.718255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.718268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.718339 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:34.718489 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719635 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.720858 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721537 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721593 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721638 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.721916 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.721960 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":92,"threads_started":1}
I20260928 22:44:34.722033 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.0.98.2:45709)
0928 22:44:34.717972 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.717977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.718107 (+ 130us) server_negotiation.cc:407] Connection header received
0928 22:44:34.718147 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.718148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.718161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.718240 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:34.718462 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719397 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721072 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721577 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721640 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721705 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.721913 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.721953 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260928 22:44:34.722280 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35267 (local address 127.0.0.1:42316)
0928 22:44:34.717951 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.717959 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.717993 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.718336 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.718337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.718339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:34.718416 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.718418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719736 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.719737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.720782 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.720784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721671 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.721731 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721735 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.721780 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.721817 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260928 22:44:34.722294 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.717586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45709 (local address 127.0.0.1:55742)
0928 22:44:34.718047 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.718054 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.718098 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.718263 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.718266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.718272 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:34.718381 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.718383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.719480 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.719482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.721010 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.721011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.721738 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.721738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.721789 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.721792 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.721833 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.721868 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":43,"thread_start_us":49,"threads_started":1}
I20260928 22:44:34.722689 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd133bbc7a4987b3e24890ac950f33 (DEFAULT_TABLE table=test-table [id=073c23de308f4b1190d3b41224435391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:34.723059 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd133bbc7a4987b3e24890ac950f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:34.722959 1221 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd133bbc7a4987b3e24890ac950f33 (DEFAULT_TABLE table=test-table [id=073c23de308f4b1190d3b41224435391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:34.723253 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd133bbc7a4987b3e24890ac950f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:34.725450 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd133bbc7a4987b3e24890ac950f33 (DEFAULT_TABLE table=test-table [id=073c23de308f4b1190d3b41224435391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:34.725806 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd133bbc7a4987b3e24890ac950f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:34.726424 1435 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Bootstrap starting.
I20260928 22:44:34.726627 1436 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: Bootstrap starting.
I20260928 22:44:34.727305 1436 tablet_bootstrap.cc:654] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:34.727890 1436 log.cc:824] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:34.728543 1436 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: No bootstrap required, opened a new log
I20260928 22:44:34.728606 1436 ts_tablet_manager.cc:1397] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:44:34.728683 1435 tablet_bootstrap.cc:654] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:34.729231 1435 log.cc:824] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:34.729856 1435 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: No bootstrap required, opened a new log
I20260928 22:44:34.729905 1435 ts_tablet_manager.cc:1397] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 22:44:34.730207 1436 raft_consensus.cc:348] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.730355 1436 raft_consensus.cc:374] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:34.730386 1436 raft_consensus.cc:729] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81bfaa0bd1f741abb2e93549497484e8, State: Initialized, Role: FOLLOWER
I20260928 22:44:34.730499 1436 consensus_queue.cc:260] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.730748 1436 ts_tablet_manager.cc:1428] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:44:34.730759 1421 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
I20260928 22:44:34.730939 1438 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: Bootstrap starting.
I20260928 22:44:34.732017 1438 tablet_bootstrap.cc:654] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:34.731962 1435 raft_consensus.cc:348] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.732105 1435 raft_consensus.cc:374] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:34.732136 1435 raft_consensus.cc:729] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5421895e62e452fb91d946e032caeff, State: Initialized, Role: FOLLOWER
I20260928 22:44:34.732244 1435 consensus_queue.cc:260] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.732494 1435 ts_tablet_manager.cc:1428] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:44:34.732501 1438 log.cc:824] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:34.732506 1287 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
I20260928 22:44:34.733234 1438 tablet_bootstrap.cc:492] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: No bootstrap required, opened a new log
I20260928 22:44:34.733305 1438 ts_tablet_manager.cc:1397] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:44:34.734647 1438 raft_consensus.cc:348] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.734778 1438 raft_consensus.cc:374] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:34.734809 1438 raft_consensus.cc:729] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2fe17e071247669154f9d56a672f3e, State: Initialized, Role: FOLLOWER
I20260928 22:44:34.734920 1438 consensus_queue.cc:260] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.735148 1438 ts_tablet_manager.cc:1428] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:44:34.735188 1153 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
I20260928 22:44:34.788601 1441 raft_consensus.cc:482] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:34.788802 1441 raft_consensus.cc:504] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
W20260928 22:44:34.789192 1288 tablet.cc:2369] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:34.789172 1441 leader_election.cc:288] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2fe17e071247669154f9d56a672f3e (127.0.98.1:35267), 81bfaa0bd1f741abb2e93549497484e8 (127.0.98.3:42663)
I20260928 22:44:34.793069 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.789501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38575 (local address 127.0.98.1:35267)
0928 22:44:34.789551 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.789553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.789562 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:34.789589 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.789590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.789603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.789654 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:34.789851 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.790658 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792127 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.792703 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792797 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.792807 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.792955 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.792997 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:44:34.793069 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.789516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36749 (local address 127.0.98.3:42663)
0928 22:44:34.789556 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.789558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.789783 (+ 225us) server_negotiation.cc:407] Connection header received
0928 22:44:34.789812 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.789813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.789825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.789866 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:34.790005 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.790700 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792126 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.792740 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792796 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.792807 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.792960 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.792997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:34.793097 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.789343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35267 (local address 127.0.98.2:38575)
0928 22:44:34.789395 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.789408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.789455 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.789676 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.789677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.789680 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:34.789767 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.789769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.790729 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.790730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.792034 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.792037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.792796 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.792887 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.792889 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.792942 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.792994 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":35}
I20260928 22:44:34.793102 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.789344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42663 (local address 127.0.98.2:36749)
0928 22:44:34.789704 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:34.789713 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:34.789760 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:34.789879 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:34.789880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:34.789881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:34.789947 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.789949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.790765 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.790766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.792034 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:34.792037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.792802 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.792802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:34.792884 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.792888 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:34.792942 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:44:34.792994 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":212,"thread_start_us":105,"threads_started":1}
I20260928 22:44:34.793468 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd133bbc7a4987b3e24890ac950f33" candidate_uuid: "b5421895e62e452fb91d946e032caeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bfaa0bd1f741abb2e93549497484e8" is_pre_election: true
I20260928 22:44:34.793644 1375 raft_consensus.cc:2454] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5421895e62e452fb91d946e032caeff in term 0.
I20260928 22:44:34.793664 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd133bbc7a4987b3e24890ac950f33" candidate_uuid: "b5421895e62e452fb91d946e032caeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2fe17e071247669154f9d56a672f3e" is_pre_election: true
I20260928 22:44:34.793800 1107 raft_consensus.cc:2454] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5421895e62e452fb91d946e032caeff in term 0.
I20260928 22:44:34.793898 1174 leader_election.cc:302] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81bfaa0bd1f741abb2e93549497484e8, b5421895e62e452fb91d946e032caeff; no voters:
I20260928 22:44:34.794014 1441 raft_consensus.cc:2781] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:34.794068 1441 raft_consensus.cc:482] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:34.794091 1441 raft_consensus.cc:3037] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:34.794744 1441 raft_consensus.cc:504] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.794863 1441 leader_election.cc:288] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [CANDIDATE]: Term 1 election: Requested vote from peers 4c2fe17e071247669154f9d56a672f3e (127.0.98.1:35267), 81bfaa0bd1f741abb2e93549497484e8 (127.0.98.3:42663)
I20260928 22:44:34.794992 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd133bbc7a4987b3e24890ac950f33" candidate_uuid: "b5421895e62e452fb91d946e032caeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2fe17e071247669154f9d56a672f3e"
I20260928 22:44:34.795017 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd133bbc7a4987b3e24890ac950f33" candidate_uuid: "b5421895e62e452fb91d946e032caeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bfaa0bd1f741abb2e93549497484e8"
I20260928 22:44:34.795061 1107 raft_consensus.cc:3037] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:34.795085 1375 raft_consensus.cc:3037] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:34.795753 1107 raft_consensus.cc:2454] T d4cd133bbc7a4987b3e24890ac950f33 P 4c2fe17e071247669154f9d56a672f3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5421895e62e452fb91d946e032caeff in term 1.
I20260928 22:44:34.795789 1375 raft_consensus.cc:2454] T d4cd133bbc7a4987b3e24890ac950f33 P 81bfaa0bd1f741abb2e93549497484e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5421895e62e452fb91d946e032caeff in term 1.
I20260928 22:44:34.795926 1177 leader_election.cc:302] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2fe17e071247669154f9d56a672f3e, b5421895e62e452fb91d946e032caeff; no voters:
I20260928 22:44:34.796020 1441 raft_consensus.cc:2781] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:34.796169 1441 raft_consensus.cc:686] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [term 1 LEADER]: Becoming Leader. State: Replica: b5421895e62e452fb91d946e032caeff, State: Running, Role: LEADER
I20260928 22:44:34.796262 1441 consensus_queue.cc:237] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } }
I20260928 22:44:34.797065 962 catalog_manager.cc:5495] T d4cd133bbc7a4987b3e24890ac950f33 P b5421895e62e452fb91d946e032caeff reported cstate change: term changed from 0 to 1, leader changed from <none> to b5421895e62e452fb91d946e032caeff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b5421895e62e452fb91d946e032caeff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2fe17e071247669154f9d56a672f3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81bfaa0bd1f741abb2e93549497484e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5421895e62e452fb91d946e032caeff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45709 } health_report { overall_health: HEALTHY } } }
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:34.816282 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.809490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.0.98.1:35267)
0928 22:44:34.809570 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.809572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.809643 (+ 71us) server_negotiation.cc:407] Connection header received
0928 22:44:34.809710 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.809711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.809722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.809762 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.810272 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.810740 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.811431 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.811603 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.811646 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.813754 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.813764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.813765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.813771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.815320 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.815627 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.815631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.815631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.815692 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.815833 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.815834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.815834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.815977 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.816083 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.816115 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.816180 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:34.837497 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.830880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.98.1:35267)
0928 22:44:34.830943 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.830945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.831009 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:44:34.831076 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.831077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.831087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.831133 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.831650 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.832122 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.832868 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.833053 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.833105 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.835192 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.835194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.835195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.835203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.836435 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.836771 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.836772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.836773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.836822 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.836980 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.836981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.836981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.837074 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.837155 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.837330 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.837405 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:34.857903 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:34.851416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.1:35267)
0928 22:44:34.851470 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:34.851472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:34.851535 (+ 63us) server_negotiation.cc:407] Connection header received
0928 22:44:34.851606 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:34.851607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:34.851617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:34.851661 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:34.852194 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.852664 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.853388 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:34.853567 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:34.853621 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:34.855682 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:34.855683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:34.855684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:34.855692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:34.856940 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.857249 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.857250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.857250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.857299 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:34.857453 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:34.857454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:34.857455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:34.857549 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:34.857633 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:34.857735 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:44:34.857810 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 22:44:34.858330 1067 tablet_service.cc:2908] Rejecting scan request for tablet d4cd133bbc7a4987b3e24890ac950f33: Uninitialized: clean time has not yet been initialized
I20260928 22:44:34.859138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1024
I20260928 22:44:34.863628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1158
I20260928 22:44:34.867931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1292
I20260928 22:44:34.871825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 930
2026-09-28T22:44:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1823 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[1459](info): set up 2 sockets
Sep 28 22:44:34 dist-test-slave-pxr1 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260928 22:44:35.812000 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:35 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:35.920707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:36.000084 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:36.000303 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:36.000324 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:36.001525 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:36.001577 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:44:36.001590 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:36.001612 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:36.001636 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:36.001657 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:36.003216 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46485
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:36.003463 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:36.003687 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:36.008322 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.008386 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.008348 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:36.008460 1472 server_base.cc:1034] running on GCE node
I20260928 22:44:36.008786 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:36.009099 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:36.010264 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790635476010248 us; error 30 us; skew 500 ppm
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:36.012678 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:36.013137 1472 webserver.cc:460] Webserver started at http://127.0.98.62:45977/ using document root <none> and password file <none>
I20260928 22:44:36.013330 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:36.013375 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:36.013465 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:36.014470 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "2fa9925676e24a44aee6a4b1c01e7af3"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "ce4408e73b33c70ef663073620e002fd"
server_key_iv: "e54c3f7af80bc35b1d51dc1462b6e9e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.014758 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fa9925676e24a44aee6a4b1c01e7af3"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "ce4408e73b33c70ef663073620e002fd"
server_key_iv: "e54c3f7af80bc35b1d51dc1462b6e9e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.015960 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:44:36.016614 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.016783 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:36.016866 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "2fa9925676e24a44aee6a4b1c01e7af3"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "ce4408e73b33c70ef663073620e002fd"
server_key_iv: "e54c3f7af80bc35b1d51dc1462b6e9e1"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.016928 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:36.037400 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:36.039552 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:36.039691 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:36.043907 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46485
I20260928 22:44:36.043959 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46485 every 8 connection(s)
I20260928 22:44:36.044273 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:36.044965 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.044978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1472
I20260928 22:44:36.045051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:36.045189 392 external_mini_cluster.cc:1428] Setting key e46e22cd1119ed24dc492d1c0aca28d7
I20260928 22:44:36.048144 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Bootstrap starting.
I20260928 22:44:36.048861 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.049124 1550 log.cc:824] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:36.049773 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: No bootstrap required, opened a new log
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:36.051846 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260928 22:44:36.051998 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.052035 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa9925676e24a44aee6a4b1c01e7af3, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.052162 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260928 22:44:36.052219 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:36.052242 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:36.052287 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.052963 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260928 22:44:36.053084 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa9925676e24a44aee6a4b1c01e7af3; no voters:
I20260928 22:44:36.053248 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:36.053298 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:36.053491 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:36.054080 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa9925676e24a44aee6a4b1c01e7af3, State: Running, Role: LEADER
I20260928 22:44:36.054170 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260928 22:44:36.054520 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa9925676e24a44aee6a4b1c01e7af3. Latest consensus state: current_term: 1 leader_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260928 22:44:36.054610 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:36.054522 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa9925676e24a44aee6a4b1c01e7af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260928 22:44:36.055055 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:36.054781 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:36.055564 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:36.055809 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.045681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.0.98.62:46485)
0928 22:44:36.045853 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.045863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.045897 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:44:36.046385 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.046396 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.046461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.046612 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.047168 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.048546 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.049370 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.049860 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.049965 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.052390 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.052395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.052396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.052408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.054263 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.054671 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.054675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.054675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.054777 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.054980 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.054981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.054981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.055246 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.055381 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.055494 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.055632 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 22:44:36.056900 1569 catalog_manager.cc:1347] Generated new cluster ID: 09ef9bc040d842a2889335f77316f113
I20260928 22:44:36.056949 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:36.074961 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:36.075891 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:36.081938 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Generated new TSK 0
I20260928 22:44:36.082168 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:36.112861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:36.193408 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:36.193616 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:36.193640 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:36.194813 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:36.194845 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:36.194864 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:36.194880 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:36.196382 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:36.196596 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:36.196837 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:36.201539 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:36.201615 1576 server_base.cc:1034] running on GCE node
W20260928 22:44:36.201555 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.201546 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:36.201915 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:36.202116 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:36.203253 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790635476203244 us; error 26 us; skew 500 ppm
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:36.205789 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:36.206198 1576 webserver.cc:460] Webserver started at http://127.0.98.1:43965/ using document root <none> and password file <none>
I20260928 22:44:36.206401 1576 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:36.206444 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:36.206558 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:36.207603 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "7b2287f727508b1d7b260090f8ff3f97"
server_key_iv: "e6658554868f2ddcee8424723d1b0795"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.207924 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "7b2287f727508b1d7b260090f8ff3f97"
server_key_iv: "e6658554868f2ddcee8424723d1b0795"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.209164 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:36.209836 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.209964 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:36.210043 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "7b2287f727508b1d7b260090f8ff3f97"
server_key_iv: "e6658554868f2ddcee8424723d1b0795"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.210103 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:36.233918 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:36.236078 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:36.236215 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:36.236698 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:36.236756 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.236794 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:36.236815 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.243081 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33725
I20260928 22:44:36.243213 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33725 every 8 connection(s)
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:36.243474 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:36.246052 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.237225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43315 (local address 127.0.98.62:46485)
0928 22:44:36.237289 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.237292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.237483 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:44:36.238045 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.238047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.238059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.238134 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.238815 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.239598 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.241147 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.241366 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.241428 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.243856 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.243858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.243858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.243865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.244970 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.245367 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.245367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.245368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.245413 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.245585 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.245585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.245586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.245683 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.245727 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.245926 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.245991 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:44:36.246223 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.237203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.1:43315)
0928 22:44:36.237365 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.237406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.237918 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.238226 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.238229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.238497 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.238720 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.238724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.239700 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.239702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.241055 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.241060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.241458 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.241459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.241591 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.242115 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.242126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.243749 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.245076 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.245082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.245084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.245305 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.245449 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.245450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.245450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.245544 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.245726 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.245729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.245841 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.245915 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.246041 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260928 22:44:36.246687 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260928 22:44:36.246764 1707 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:36.246919 1707 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260928 22:44:36.247303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1576
I20260928 22:44:36.247387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:36.247466 1503 ts_manager.cc:194] Registered new tserver with Master: ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725)
I20260928 22:44:36.247519 392 external_mini_cluster.cc:1428] Setting key 5108addd0d7aa137510c2abad2d515bd
I20260928 22:44:36.248327 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43315
I20260928 22:44:36.252887 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.247724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37333 (local address 127.0.98.62:46485)
0928 22:44:36.247778 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.247780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.247949 (+ 169us) server_negotiation.cc:407] Connection header received
0928 22:44:36.247998 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.247999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.248008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.248051 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.248357 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.248941 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.249625 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.249815 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.249869 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.250684 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.250686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.250686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.250692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.251918 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.252207 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.252209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.252209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.252258 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.252412 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.252413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.252414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.252518 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.252571 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.252703 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.252795 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:44:36.252897 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.247622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.1:37333)
0928 22:44:36.247885 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.247894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.247945 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.248053 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.248053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.248217 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.248275 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.248277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.249044 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.249045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.249559 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.249561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.249873 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.249874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.249935 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.250327 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.250331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.250606 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.251994 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.251995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.251995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.252149 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.252309 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.252309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.252310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.252364 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.252570 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.252571 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.252615 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.252674 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.252793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":207,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:36.269744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:36.348455 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:36.348660 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:36.348683 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:36.349862 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:36.349903 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:36.349923 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:36.349939 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:36.351413 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:36.351593 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:36.351769 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:36.356770 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.356840 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.356770 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:36.356787 1716 server_base.cc:1034] running on GCE node
I20260928 22:44:36.357105 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:36.357343 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:36.358489 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790635476358479 us; error 26 us; skew 500 ppm
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:36.360927 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:36.361356 1716 webserver.cc:460] Webserver started at http://127.0.98.2:35731/ using document root <none> and password file <none>
I20260928 22:44:36.361545 1716 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:36.361589 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:36.361711 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:36.362735 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3ee90be8c414f4cb7349838e527871b"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "d3316ce7fcb8b0baeb82ff41fe1019ba"
server_key_iv: "a316dfe2601ec68101a300e2f64168b3"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.363050 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3ee90be8c414f4cb7349838e527871b"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "d3316ce7fcb8b0baeb82ff41fe1019ba"
server_key_iv: "a316dfe2601ec68101a300e2f64168b3"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.364287 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:36.364941 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.365074 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.365166 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "f3ee90be8c414f4cb7349838e527871b"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "d3316ce7fcb8b0baeb82ff41fe1019ba"
server_key_iv: "a316dfe2601ec68101a300e2f64168b3"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.365233 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:36.376174 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:36.378322 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:36.378434 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:36.378893 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:36.378932 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.378959 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:36.378975 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.385152 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41575
I20260928 22:44:36.385212 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41575 every 8 connection(s)
I20260928 22:44:36.385495 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:36.388285 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.379284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57585 (local address 127.0.98.62:46485)
0928 22:44:36.379323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.379325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.379522 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:44:36.380008 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.380009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.380024 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.380098 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.380892 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.381648 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.383131 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.383425 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.383469 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.386044 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.386045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.386045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.386052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.387084 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.387423 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.387425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.387425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.387487 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.387697 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.387698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.387698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.387813 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.387867 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.388112 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.388197 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:36.388474 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.379295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.2:57585)
0928 22:44:36.379417 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.379451 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.379915 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.380174 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.380177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.380479 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.380797 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.380803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.381734 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.381734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.383027 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.383031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.383565 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.383566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.383714 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.384332 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.384342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.385959 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.387153 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.387159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.387161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.387361 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.387540 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.387541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.387541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.387644 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.387864 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.387869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.388002 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.388114 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.388269 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260928 22:44:36.388958 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260928 22:44:36.389035 1847 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:36.389199 1847 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260928 22:44:36.389552 1504 ts_manager.cc:194] Registered new tserver with Master: f3ee90be8c414f4cb7349838e527871b (127.0.98.2:41575)
I20260928 22:44:36.389892 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57585
I20260928 22:44:36.394050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1716
I20260928 22:44:36.394157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:36.394320 392 external_mini_cluster.cc:1428] Setting key f91b46cdd6929a90c1a8d56bd43a3390
I20260928 22:44:36.394524 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.389931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45547 (local address 127.0.98.62:46485)
0928 22:44:36.390004 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.390006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.390112 (+ 106us) server_negotiation.cc:407] Connection header received
0928 22:44:36.390147 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.390148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.390157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.390207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.390506 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.390945 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.391544 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.391710 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.391749 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.392442 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.392443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.392443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.392449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.393545 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.393816 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.393816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.393817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.393859 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.394016 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.394016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.394017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.394147 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.394200 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.394368 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.394436 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:44:36.394532 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.389870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.2:45547)
0928 22:44:36.390051 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.390059 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.390108 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.390212 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.390213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.390395 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.390446 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.390447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.391013 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.391013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.391491 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.391492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.391755 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.391756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.391796 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.392133 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.392136 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.392379 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.393612 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.393613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.393613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.393763 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.393904 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.393905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.393905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.393963 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.394230 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.394231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.394275 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.394330 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.394430 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:36.415382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:36.493237 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:36.493441 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:36.493465 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:36.494622 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:36.494663 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:36.494681 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:36.494699 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:36.496166 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:36.496380 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:36.496590 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:36.501241 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.501327 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:36.501341 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:36.501418 1856 server_base.cc:1034] running on GCE node
I20260928 22:44:36.501616 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:36.501837 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:36.502976 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790635476502960 us; error 27 us; skew 500 ppm
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:36.505267 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:36.505734 1856 webserver.cc:460] Webserver started at http://127.0.98.3:37991/ using document root <none> and password file <none>
I20260928 22:44:36.505947 1856 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:36.505995 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:36.506119 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:36.507141 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "c383b7ca017f42de94e930eabeccd906"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "28d81191062c9c35a7705caba99c12f6"
server_key_iv: "4892e7425b1ffdb4b792013595e33afc"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.507458 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c383b7ca017f42de94e930eabeccd906"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "28d81191062c9c35a7705caba99c12f6"
server_key_iv: "4892e7425b1ffdb4b792013595e33afc"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.508649 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:44:36.509354 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.509538 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.509797 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "c383b7ca017f42de94e930eabeccd906"
format_stamp: "Formatted at 2026-09-28 22:44:36 on dist-test-slave-pxr1"
server_key: "28d81191062c9c35a7705caba99c12f6"
server_key_iv: "4892e7425b1ffdb4b792013595e33afc"
server_key_version: "encryptionkey@0"
I20260928 22:44:36.509867 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:36.523159 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:36.525151 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:36.525278 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:36.525763 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:36.525804 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.525835 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:36.525861 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:36.532534 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38757
I20260928 22:44:36.532598 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38757 every 8 connection(s)
I20260928 22:44:36.533007 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:36.535306 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.526299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40761 (local address 127.0.98.62:46485)
0928 22:44:36.526364 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.526367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.526576 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:44:36.527257 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.527258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.527268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.527306 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.528142 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.528894 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.530382 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.530583 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.530654 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.533046 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.533047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.533048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.533057 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.534285 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.534624 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.534625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.534625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.534670 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.534832 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.534832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.534833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.534925 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.534969 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.535187 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.535244 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:44:36.535473 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.526279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.3:40761)
0928 22:44:36.526433 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.526480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.527140 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.527390 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.527396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.527740 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.528034 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.528040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.528970 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.528971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.530273 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.530277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.530661 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.530662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.530796 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.531330 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.531339 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.532957 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.534353 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.534357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.534359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.534564 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.534710 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.534711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.534711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.534795 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.534967 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.534970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.535093 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.535171 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.535287 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 22:44:36.535902 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260928 22:44:36.535976 1987 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:36.536137 1987 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260928 22:44:36.536526 1503 ts_manager.cc:194] Registered new tserver with Master: c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757)
I20260928 22:44:36.537006 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40761
I20260928 22:44:36.539584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 1856
I20260928 22:44:36.539666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:36.539814 392 external_mini_cluster.cc:1428] Setting key 02f23bbb2c06b61f8d5a768183b638dc
I20260928 22:44:36.540817 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:44:36.542199 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.3:46119)
0928 22:44:36.536939 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.536949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.537007 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.537165 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.537166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.537355 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:36.537418 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.537419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.538043 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.538044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.538549 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.538550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.538871 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.538872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.538928 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.539330 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:36.539335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:36.539604 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:36.541390 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.541391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.541391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.541549 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.541729 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:36.541729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:36.541729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:36.541772 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:36.541967 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:36.541967 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:36.542002 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.542045 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.542127 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 22:44:36.542218 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.536845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46119 (local address 127.0.98.62:46485)
0928 22:44:36.536884 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.536886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.536995 (+ 109us) server_negotiation.cc:407] Connection header received
0928 22:44:36.537058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.537058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.537068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.537124 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.537489 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.537959 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.538622 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.538816 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.538868 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.539717 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.539719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.539719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.539726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.541316 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.541614 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.541615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.541615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.541670 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.541806 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.541807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.541807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.541920 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.541975 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.542069 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.542137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:44:36 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:36.570060 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.563183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.0.98.62:46485)
0928 22:44:36.563242 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.563244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.563255 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:36.563285 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.563286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.563297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.563371 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:36.563928 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.564671 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.565430 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.565630 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.565693 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.567830 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:36.567832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:36.567832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:36.567841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:36.569031 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.569346 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.569347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.569348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.569402 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:36.569568 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:36.569568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:36.569569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:36.569690 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:36.569767 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.569889 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.569967 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:44:36.571449 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:36.573275 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:36.579402 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.0.98.2:41575)
0928 22:44:36.575713 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.575717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.575728 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:36.575767 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.575769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.575785 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.575866 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:36.576014 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.577294 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.578507 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579010 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579074 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579170 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.579266 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.579303 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":72,"threads_started":1}
I20260928 22:44:36.579485 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.0.1:32812)
0928 22:44:36.575650 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.575658 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.575696 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.576074 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.576075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.576076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:36.576119 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.576120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.577023 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.577024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.578382 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.578385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579245 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.579309 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579313 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.579342 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.579385 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":43,"thread_start_us":48,"threads_started":1}
I20260928 22:44:36.579386 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41575 (local address 127.0.0.1:59486)
0928 22:44:36.575450 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.575489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.575556 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.575879 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.575881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.575884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:36.575955 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.575957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.577369 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.577370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.578446 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.578448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579087 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.579165 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579180 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.579245 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.579303 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260928 22:44:36.579559 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.98.3:38757)
0928 22:44:36.575549 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.575554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.575816 (+ 262us) server_negotiation.cc:407] Connection header received
0928 22:44:36.575856 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.575859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.575873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.575935 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:36.576149 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.577294 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.578366 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579055 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579136 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.579426 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.579466 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260928 22:44:36.579676 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.0.98.1:33725)
0928 22:44:36.575805 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.575809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.575816 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:36.575848 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.575850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.575865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.576080 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:36.576190 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.576948 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.578466 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579152 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579428 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579508 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.579532 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.579572 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 22:44:36.579726 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.575290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38757 (local address 127.0.0.1:60404)
0928 22:44:36.575553 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.575564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.575625 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.575953 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.575954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.575956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:36.576010 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.576012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.577404 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.577405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.578293 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.578295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.579242 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.579243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.579310 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.579315 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.579357 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.579389 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260928 22:44:36.580492 1781 tablet_service.cc:1467] Processing CreateTablet for tablet bc770cb48dcd4477af554e264dfb8e90 (DEFAULT_TABLE table=test-table [id=f5791b9068eb46dcac9dadbd041ebf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.580482 1641 tablet_service.cc:1467] Processing CreateTablet for tablet bc770cb48dcd4477af554e264dfb8e90 (DEFAULT_TABLE table=test-table [id=f5791b9068eb46dcac9dadbd041ebf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.580482 1921 tablet_service.cc:1467] Processing CreateTablet for tablet bc770cb48dcd4477af554e264dfb8e90 (DEFAULT_TABLE table=test-table [id=f5791b9068eb46dcac9dadbd041ebf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.580870 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc770cb48dcd4477af554e264dfb8e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.580871 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc770cb48dcd4477af554e264dfb8e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.580871 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc770cb48dcd4477af554e264dfb8e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.583870 2007 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Bootstrap starting.
I20260928 22:44:36.584302 2008 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Bootstrap starting.
I20260928 22:44:36.584520 2009 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Bootstrap starting.
I20260928 22:44:36.584559 2007 tablet_bootstrap.cc:654] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.584879 2007 log.cc:824] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:36.584911 2008 tablet_bootstrap.cc:654] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.585140 2009 tablet_bootstrap.cc:654] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.585194 2008 log.cc:824] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:36.585454 2009 log.cc:824] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:36.585634 2007 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: No bootstrap required, opened a new log
I20260928 22:44:36.585692 2007 ts_tablet_manager.cc:1397] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:44:36.585841 2008 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: No bootstrap required, opened a new log
I20260928 22:44:36.585901 2008 ts_tablet_manager.cc:1397] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:36.586228 2009 tablet_bootstrap.cc:492] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: No bootstrap required, opened a new log
I20260928 22:44:36.586294 2009 ts_tablet_manager.cc:1397] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:36.587421 2007 raft_consensus.cc:348] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.587553 2007 raft_consensus.cc:374] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.587584 2007 raft_consensus.cc:729] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ee90be8c414f4cb7349838e527871b, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.587699 2007 consensus_queue.cc:260] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.587765 2008 raft_consensus.cc:348] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.587924 2008 raft_consensus.cc:374] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.587967 2008 raft_consensus.cc:729] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c383b7ca017f42de94e930eabeccd906, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.587965 2007 ts_tablet_manager.cc:1428] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:44:36.588128 1847 heartbeater.cc:503] Master 127.0.98.62:46485 was elected leader, sending a full tablet report...
I20260928 22:44:36.588073 2008 consensus_queue.cc:260] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.588346 2008 ts_tablet_manager.cc:1428] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:44:36.588361 1987 heartbeater.cc:503] Master 127.0.98.62:46485 was elected leader, sending a full tablet report...
I20260928 22:44:36.588209 2009 raft_consensus.cc:348] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.588641 2009 raft_consensus.cc:374] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.588735 2009 raft_consensus.cc:729] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8c885b3a514f85a6bdc5fe55a9a3ca, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.588863 2009 consensus_queue.cc:260] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.589103 2009 ts_tablet_manager.cc:1428] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 22:44:36.589208 1707 heartbeater.cc:503] Master 127.0.98.62:46485 was elected leader, sending a full tablet report...
W20260928 22:44:36.636066 1848 tablet.cc:2369] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:36.648589 2013 raft_consensus.cc:482] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:36.648738 2013 raft_consensus.cc:504] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.649106 2013 leader_election.cc:288] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757), ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725)
I20260928 22:44:36.651765 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.649294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38757 (local address 127.0.98.2:54093)
0928 22:44:36.649354 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.649366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.649421 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.649560 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.649561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.649563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:36.649627 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.649629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.650239 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.650240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.650989 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.650990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.651513 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.651588 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.651591 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.651637 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.651679 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:44:36.651757 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.649380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54093 (local address 127.0.98.3:38757)
0928 22:44:36.649440 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.649442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.649468 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:44:36.649495 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.649496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.649508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.649553 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:36.649694 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.650163 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651082 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.651441 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651501 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.651512 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.651659 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.651694 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:44:36.651892 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.649497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40565 (local address 127.0.98.1:33725)
0928 22:44:36.649541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.649543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.649627 (+ 84us) server_negotiation.cc:407] Connection header received
0928 22:44:36.649669 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.649670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.649681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.649724 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:36.649862 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.650314 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651218 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.651582 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651650 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.651659 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.651797 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.651832 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:44:36.651916 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.649387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.98.2:40565)
0928 22:44:36.649564 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:36.649572 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:36.649618 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:36.649739 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:36.649739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:36.649741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:36.649808 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.649810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.650386 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.650387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.651130 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:36.651132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.651665 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.651666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:36.651737 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.651740 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:36.651785 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:44:36.651822 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260928 22:44:36.652107 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc770cb48dcd4477af554e264dfb8e90" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c383b7ca017f42de94e930eabeccd906" is_pre_election: true
I20260928 22:44:36.652287 1941 raft_consensus.cc:2454] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 0.
I20260928 22:44:36.652518 1733 leader_election.cc:302] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c383b7ca017f42de94e930eabeccd906, f3ee90be8c414f4cb7349838e527871b; no voters:
I20260928 22:44:36.652631 2013 raft_consensus.cc:2781] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:36.652679 2013 raft_consensus.cc:482] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:36.652634 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc770cb48dcd4477af554e264dfb8e90" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" is_pre_election: true
I20260928 22:44:36.652704 2013 raft_consensus.cc:3037] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.652803 1661 raft_consensus.cc:2454] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 0.
I20260928 22:44:36.653766 2013 raft_consensus.cc:504] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.653882 2013 leader_election.cc:288] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 election: Requested vote from peers c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757), ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725)
I20260928 22:44:36.654069 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc770cb48dcd4477af554e264dfb8e90" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
I20260928 22:44:36.654036 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc770cb48dcd4477af554e264dfb8e90" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c383b7ca017f42de94e930eabeccd906"
I20260928 22:44:36.654145 1661 raft_consensus.cc:3037] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.654146 1941 raft_consensus.cc:3037] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.655045 1661 raft_consensus.cc:2454] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 1.
I20260928 22:44:36.655045 1941 raft_consensus.cc:2454] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 1.
I20260928 22:44:36.655210 1733 leader_election.cc:302] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8c885b3a514f85a6bdc5fe55a9a3ca, f3ee90be8c414f4cb7349838e527871b; no voters:
I20260928 22:44:36.655299 2013 raft_consensus.cc:2781] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:36.655472 2013 raft_consensus.cc:686] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 1 LEADER]: Becoming Leader. State: Replica: f3ee90be8c414f4cb7349838e527871b, State: Running, Role: LEADER
I20260928 22:44:36.655586 2013 consensus_queue.cc:237] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:36.656418 1503 catalog_manager.cc:5495] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ee90be8c414f4cb7349838e527871b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f3ee90be8c414f4cb7349838e527871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:36.670953 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:36.668603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.98.2:41575)
0928 22:44:36.668654 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:36.668657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:36.668666 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:36.668783 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:36.668784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:36.668805 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:36.668859 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:36.668999 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.669458 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.670229 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:36.670412 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:36.670465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:36.670654 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:36.670746 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:36.670810 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:36.670860 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 22:44:36.670881 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:44:36.672628 1941 raft_consensus.cc:1261] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Refusing update from remote peer f3ee90be8c414f4cb7349838e527871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:36.672642 1661 raft_consensus.cc:1261] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Refusing update from remote peer f3ee90be8c414f4cb7349838e527871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:36.672997 2013 consensus_queue.cc:1035] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:36.673141 2017 consensus_queue.cc:1035] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:36.675495 2020 mvcc.cc:205] Tried to move back new op lower bound from 7334442912448589824 to 7334442912382058496. Current Snapshot: MvccSnapshot[applied={T|T < 7334442912448589824}]
I20260928 22:44:36.676316 2021 mvcc.cc:205] Tried to move back new op lower bound from 7334442912448589824 to 7334442912382058496. Current Snapshot: MvccSnapshot[applied={T|T < 7334442912448589824 or (T in {7334442912448589824})}]
I20260928 22:44:36.679033 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:36.679144 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:36.679863 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Sending DeleteTablet for 3 replicas of tablet bc770cb48dcd4477af554e264dfb8e90
I20260928 22:44:36.680176 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet bc770cb48dcd4477af554e264dfb8e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59486
I20260928 22:44:36.680192 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet bc770cb48dcd4477af554e264dfb8e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260928 22:44:36.680215 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet bc770cb48dcd4477af554e264dfb8e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60404
I20260928 22:44:36.680505 2032 tablet_replica.cc:330] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: stopping tablet replica
I20260928 22:44:36.680603 2030 tablet_replica.cc:330] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: stopping tablet replica
I20260928 22:44:36.680684 2030 raft_consensus.cc:2229] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:36.680676 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 22:44:36.680830 2030 raft_consensus.cc:2258] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:44:36.680845 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:36.681164 2032 raft_consensus.cc:2229] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:36.681295 2030 ts_tablet_manager.cc:1905] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:36.682080 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 512c9d3f212e42199cc0a264e2d6059b (DEFAULT_TABLE table=test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.682099 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 512c9d3f212e42199cc0a264e2d6059b (DEFAULT_TABLE table=test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.682209 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512c9d3f212e42199cc0a264e2d6059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.682245 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512c9d3f212e42199cc0a264e2d6059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.682507 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 512c9d3f212e42199cc0a264e2d6059b (DEFAULT_TABLE table=test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:36.682590 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512c9d3f212e42199cc0a264e2d6059b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:36.683633 2008 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Bootstrap starting.
I20260928 22:44:36.683781 2009 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Bootstrap starting.
I20260928 22:44:36.683780 2007 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Bootstrap starting.
I20260928 22:44:36.684026 2008 tablet_bootstrap.cc:654] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.684152 2009 tablet_bootstrap.cc:654] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.684163 2007 tablet_bootstrap.cc:654] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:36.684579 2008 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: No bootstrap required, opened a new log
I20260928 22:44:36.684631 2008 ts_tablet_manager.cc:1397] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:36.684782 2008 raft_consensus.cc:348] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.684823 2032 raft_consensus.cc:2258] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:36.684852 2008 raft_consensus.cc:374] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.684873 2008 raft_consensus.cc:729] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c383b7ca017f42de94e930eabeccd906, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.684916 2008 consensus_queue.cc:260] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.684996 2008 ts_tablet_manager.cc:1428] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:36.685201 2009 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: No bootstrap required, opened a new log
I20260928 22:44:36.685246 2009 ts_tablet_manager.cc:1397] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:44:36.685387 2009 raft_consensus.cc:348] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.685473 2009 raft_consensus.cc:374] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.685480 2030 ts_tablet_manager.cc:1918] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:36.685495 2009 raft_consensus.cc:729] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8c885b3a514f85a6bdc5fe55a9a3ca, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.685494 2032 ts_tablet_manager.cc:1905] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:36.685544 2030 log.cc:1196] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/bc770cb48dcd4477af554e264dfb8e90
I20260928 22:44:36.685541 2009 consensus_queue.cc:260] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.685624 2009 ts_tablet_manager.cc:1428] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.685822 2030 ts_tablet_manager.cc:1939] T bc770cb48dcd4477af554e264dfb8e90 P f3ee90be8c414f4cb7349838e527871b: Deleting consensus metadata
I20260928 22:44:36.686213 1488 catalog_manager.cc:4841] TS f3ee90be8c414f4cb7349838e527871b (127.0.98.2:41575): tablet bc770cb48dcd4477af554e264dfb8e90 (table test-table [id=f5791b9068eb46dcac9dadbd041ebf11]) successfully deleted
I20260928 22:44:36.686280 2031 tablet_replica.cc:330] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: stopping tablet replica
I20260928 22:44:36.686388 2031 raft_consensus.cc:2229] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:36.686486 2031 raft_consensus.cc:2258] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:36.686862 2031 ts_tablet_manager.cc:1905] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:36.686894 2032 ts_tablet_manager.cc:1918] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:36.686909 2007 tablet_bootstrap.cc:492] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: No bootstrap required, opened a new log
I20260928 22:44:36.686945 2007 ts_tablet_manager.cc:1397] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 22:44:36.686954 2032 log.cc:1196] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/bc770cb48dcd4477af554e264dfb8e90
I20260928 22:44:36.687120 2007 raft_consensus.cc:348] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.687194 2007 raft_consensus.cc:374] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:36.687214 2007 raft_consensus.cc:729] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ee90be8c414f4cb7349838e527871b, State: Initialized, Role: FOLLOWER
I20260928 22:44:36.687211 2032 ts_tablet_manager.cc:1939] T bc770cb48dcd4477af554e264dfb8e90 P c383b7ca017f42de94e930eabeccd906: Deleting consensus metadata
I20260928 22:44:36.687258 2007 consensus_queue.cc:260] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.687352 2007 ts_tablet_manager.cc:1428] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:36.687620 1489 catalog_manager.cc:4841] TS c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757): tablet bc770cb48dcd4477af554e264dfb8e90 (table test-table [id=f5791b9068eb46dcac9dadbd041ebf11]) successfully deleted
I20260928 22:44:36.688345 2031 ts_tablet_manager.cc:1918] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:36.688431 2031 log.cc:1196] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/bc770cb48dcd4477af554e264dfb8e90
I20260928 22:44:36.688633 2031 ts_tablet_manager.cc:1939] T bc770cb48dcd4477af554e264dfb8e90 P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting consensus metadata
I20260928 22:44:36.689009 1489 catalog_manager.cc:4841] TS ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725): tablet bc770cb48dcd4477af554e264dfb8e90 (table test-table [id=f5791b9068eb46dcac9dadbd041ebf11]) successfully deleted
W20260928 22:44:36.744215 1708 tablet.cc:2369] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:44:36.783535 1988 tablet.cc:2369] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:36.982064 2025 raft_consensus.cc:482] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:36.982178 2025 raft_consensus.cc:504] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.982338 2025 leader_election.cc:288] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725), c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757)
I20260928 22:44:36.982590 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512c9d3f212e42199cc0a264e2d6059b" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" is_pre_election: true
I20260928 22:44:36.982594 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512c9d3f212e42199cc0a264e2d6059b" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c383b7ca017f42de94e930eabeccd906" is_pre_election: true
I20260928 22:44:36.982704 1661 raft_consensus.cc:2454] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 0.
I20260928 22:44:36.982704 1941 raft_consensus.cc:2454] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 0.
I20260928 22:44:36.982872 1733 leader_election.cc:302] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8c885b3a514f85a6bdc5fe55a9a3ca, f3ee90be8c414f4cb7349838e527871b; no voters:
I20260928 22:44:36.982971 2025 raft_consensus.cc:2781] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:36.983012 2025 raft_consensus.cc:482] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:36.983026 2025 raft_consensus.cc:3037] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.983716 2025 raft_consensus.cc:504] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.983829 2025 leader_election.cc:288] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 election: Requested vote from peers ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725), c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757)
I20260928 22:44:36.984005 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512c9d3f212e42199cc0a264e2d6059b" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
I20260928 22:44:36.984077 1661 raft_consensus.cc:3037] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.984035 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512c9d3f212e42199cc0a264e2d6059b" candidate_uuid: "f3ee90be8c414f4cb7349838e527871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c383b7ca017f42de94e930eabeccd906"
I20260928 22:44:36.984143 1941 raft_consensus.cc:3037] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:36.984663 1661 raft_consensus.cc:2454] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 1.
I20260928 22:44:36.984835 1941 raft_consensus.cc:2454] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ee90be8c414f4cb7349838e527871b in term 1.
I20260928 22:44:36.984848 1733 leader_election.cc:302] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8c885b3a514f85a6bdc5fe55a9a3ca, f3ee90be8c414f4cb7349838e527871b; no voters:
I20260928 22:44:36.984941 2025 raft_consensus.cc:2781] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:36.985008 2025 raft_consensus.cc:686] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 1 LEADER]: Becoming Leader. State: Replica: f3ee90be8c414f4cb7349838e527871b, State: Running, Role: LEADER
I20260928 22:44:36.985055 2025 consensus_queue.cc:237] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } }
I20260928 22:44:36.985626 1504 catalog_manager.cc:5495] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ee90be8c414f4cb7349838e527871b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f3ee90be8c414f4cb7349838e527871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } health_report { overall_health: UNKNOWN } } }
Sep 28 22:44:37 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:37.063581 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.057375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.0.98.62:46485)
0928 22:44:37.057442 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.057444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.057536 (+ 92us) server_negotiation.cc:407] Connection header received
0928 22:44:37.057612 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.057612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.057621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.057670 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:37.058122 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.058610 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.059275 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.059447 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.059503 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.061459 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:37.061460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:37.061461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:37.061467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:37.062729 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.063023 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.063024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.063024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.063069 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.063207 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.063208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.063208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.063302 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:37.063344 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.063451 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.063516 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:44:37.063676 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.057317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.0.1:45208)
0928 22:44:37.057469 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.057479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.057544 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.057678 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.057678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.057995 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:37.058043 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.058046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.058691 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.058692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.059207 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.059208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.059518 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.059519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.059565 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.060032 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:37.060036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:37.061381 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:37.062792 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:37.062796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:37.062798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:37.062973 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:37.063115 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:37.063116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:37.063116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:37.063168 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:37.063341 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:37.063345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:37.063389 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.063433 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.063569 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":53,"thread_start_us":89,"threads_started":1}
I20260928 22:44:37.065526 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45208:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:44:37.065732 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:37.067013 1921 tablet_service.cc:1467] Processing CreateTablet for tablet a89b44f11ab249108c39040146bd4b6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6d8381d3b444d63926599ceff3bb943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:37.067000 1781 tablet_service.cc:1467] Processing CreateTablet for tablet a89b44f11ab249108c39040146bd4b6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6d8381d3b444d63926599ceff3bb943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:37.067058 1641 tablet_service.cc:1467] Processing CreateTablet for tablet a89b44f11ab249108c39040146bd4b6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6d8381d3b444d63926599ceff3bb943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:37.067166 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89b44f11ab249108c39040146bd4b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:37.067169 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89b44f11ab249108c39040146bd4b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:37.067167 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89b44f11ab249108c39040146bd4b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:37.068368 2008 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: Bootstrap starting.
I20260928 22:44:37.068611 2009 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: Bootstrap starting.
I20260928 22:44:37.068605 2007 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: Bootstrap starting.
I20260928 22:44:37.068779 2008 tablet_bootstrap.cc:654] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:37.069012 2007 tablet_bootstrap.cc:654] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:37.069123 2009 tablet_bootstrap.cc:654] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:37.069500 2007 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: No bootstrap required, opened a new log
I20260928 22:44:37.069547 2007 ts_tablet_manager.cc:1397] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:37.069669 2009 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: No bootstrap required, opened a new log
I20260928 22:44:37.069705 2009 ts_tablet_manager.cc:1397] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:37.069664 2007 raft_consensus.cc:348] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.069725 2008 tablet_bootstrap.cc:492] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: No bootstrap required, opened a new log
I20260928 22:44:37.069732 2007 raft_consensus.cc:374] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:37.069751 2007 raft_consensus.cc:729] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ee90be8c414f4cb7349838e527871b, State: Initialized, Role: FOLLOWER
I20260928 22:44:37.069761 2008 ts_tablet_manager.cc:1397] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:37.069797 2007 consensus_queue.cc:260] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.069814 2009 raft_consensus.cc:348] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.069873 2009 raft_consensus.cc:374] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:37.069890 2009 raft_consensus.cc:729] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8c885b3a514f85a6bdc5fe55a9a3ca, State: Initialized, Role: FOLLOWER
I20260928 22:44:37.069882 2007 ts_tablet_manager.cc:1428] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:37.069891 2008 raft_consensus.cc:348] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.069924 2009 consensus_queue.cc:260] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.069950 2008 raft_consensus.cc:374] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:37.069967 2008 raft_consensus.cc:729] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c383b7ca017f42de94e930eabeccd906, State: Initialized, Role: FOLLOWER
I20260928 22:44:37.070003 2009 ts_tablet_manager.cc:1428] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:37.070007 2008 consensus_queue.cc:260] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.070096 2008 ts_tablet_manager.cc:1428] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:37.070142 2014 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.070180 2025 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.070236 2014 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.070251 2025 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.070379 2015 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.070454 2015 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.098286 2014 raft_consensus.cc:482] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:37.098389 2014 raft_consensus.cc:504] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.098707 2014 leader_election.cc:288] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3ee90be8c414f4cb7349838e527871b (127.0.98.2:41575), ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725)
I20260928 22:44:37.102447 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.098922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48953 (local address 127.0.98.1:33725)
0928 22:44:37.098966 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.098969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.099050 (+ 81us) server_negotiation.cc:407] Connection header received
0928 22:44:37.099112 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.099112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.099137 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.099203 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:37.099354 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.100098 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.101574 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.102113 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.102185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.102196 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.102339 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.102376 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:37.102456 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.098917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46425 (local address 127.0.98.2:41575)
0928 22:44:37.098966 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.098969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.099050 (+ 81us) server_negotiation.cc:407] Connection header received
0928 22:44:37.099124 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.099125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.099134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.099203 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:37.099355 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.100097 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.101573 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.102114 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.102185 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.102196 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.102339 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.102375 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:44:37.102490 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.098850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41575 (local address 127.0.98.3:46425)
0928 22:44:37.098981 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.098993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.099055 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.099214 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.099214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.099216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:37.099287 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.099289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.100177 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.100178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.101497 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.101500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.102192 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.102193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.102274 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.102279 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.102329 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.102377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":78,"thread_start_us":60,"threads_started":1}
I20260928 22:44:37.102490 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.098850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.98.3:48953)
0928 22:44:37.098994 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.099002 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.099055 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.099214 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.099214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.099217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:37.099296 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.099297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.100176 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.100178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.101497 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.101500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.102190 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.102191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.102274 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.102278 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.102329 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.102377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":30,"thread_start_us":49,"threads_started":1}
I20260928 22:44:37.102767 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89b44f11ab249108c39040146bd4b6d" candidate_uuid: "c383b7ca017f42de94e930eabeccd906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ee90be8c414f4cb7349838e527871b" is_pre_election: true
I20260928 22:44:37.102825 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89b44f11ab249108c39040146bd4b6d" candidate_uuid: "c383b7ca017f42de94e930eabeccd906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" is_pre_election: true
I20260928 22:44:37.102905 1661 raft_consensus.cc:2454] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c383b7ca017f42de94e930eabeccd906 in term 0.
I20260928 22:44:37.102902 1801 raft_consensus.cc:2454] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c383b7ca017f42de94e930eabeccd906 in term 0.
I20260928 22:44:37.103080 1873 leader_election.cc:302] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8c885b3a514f85a6bdc5fe55a9a3ca, c383b7ca017f42de94e930eabeccd906; no voters:
I20260928 22:44:37.103212 2014 raft_consensus.cc:2781] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:37.103266 2014 raft_consensus.cc:482] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:37.103291 2014 raft_consensus.cc:3037] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:37.103916 2014 raft_consensus.cc:504] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.104031 2014 leader_election.cc:288] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [CANDIDATE]: Term 1 election: Requested vote from peers f3ee90be8c414f4cb7349838e527871b (127.0.98.2:41575), ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725)
I20260928 22:44:37.104187 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89b44f11ab249108c39040146bd4b6d" candidate_uuid: "c383b7ca017f42de94e930eabeccd906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca"
I20260928 22:44:37.104208 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89b44f11ab249108c39040146bd4b6d" candidate_uuid: "c383b7ca017f42de94e930eabeccd906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ee90be8c414f4cb7349838e527871b"
I20260928 22:44:37.104261 1661 raft_consensus.cc:3037] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:37.104271 1801 raft_consensus.cc:3037] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:37.104987 1661 raft_consensus.cc:2454] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c383b7ca017f42de94e930eabeccd906 in term 1.
I20260928 22:44:37.105037 1801 raft_consensus.cc:2454] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c383b7ca017f42de94e930eabeccd906 in term 1.
I20260928 22:44:37.105165 1873 leader_election.cc:302] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8c885b3a514f85a6bdc5fe55a9a3ca, c383b7ca017f42de94e930eabeccd906; no voters:
I20260928 22:44:37.105285 2014 raft_consensus.cc:2781] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:37.105393 2014 raft_consensus.cc:686] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [term 1 LEADER]: Becoming Leader. State: Replica: c383b7ca017f42de94e930eabeccd906, State: Running, Role: LEADER
I20260928 22:44:37.105450 2014 consensus_queue.cc:237] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } }
I20260928 22:44:37.105769 2048 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: TxnStatusTablet state changed. Reason: New leader c383b7ca017f42de94e930eabeccd906. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.105922 2048 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:37.106108 2050 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:44:37.106189 2050 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:44:37.106128 1504 catalog_manager.cc:5495] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 reported cstate change: term changed from 0 to 1, leader changed from <none> to c383b7ca017f42de94e930eabeccd906 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:37.117285 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.114594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.0.98.3:38757)
0928 22:44:37.114644 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.114646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.114658 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:44:37.114746 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.114746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.114761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.114818 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:37.114975 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.115433 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.116533 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.116743 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.116807 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.116988 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.117082 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.117128 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.117191 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.117208 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:44:37.117317 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.114537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38757 (local address 127.0.0.1:60418)
0928 22:44:37.114623 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.114634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.114677 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.114840 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.114841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.114843 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:37.114897 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.114900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.115497 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.115498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.116448 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.116451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.116808 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.116810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.116880 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.116901 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.117134 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.117139 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.117187 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.117236 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:44:37.118345 1661 raft_consensus.cc:1261] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Refusing update from remote peer c383b7ca017f42de94e930eabeccd906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:37.118412 1801 raft_consensus.cc:1261] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Refusing update from remote peer c383b7ca017f42de94e930eabeccd906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:37.118610 2014 consensus_queue.cc:1035] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:37.118706 2048 consensus_queue.cc:1035] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:37.119206 2015 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: TxnStatusTablet state changed. Reason: New leader c383b7ca017f42de94e930eabeccd906. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.119295 2015 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.119311 2025 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: TxnStatusTablet state changed. Reason: New leader c383b7ca017f42de94e930eabeccd906. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.119398 2025 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.120162 2048 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.120241 2048 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:37.120366 2022 mvcc.cc:205] Tried to move back new op lower bound from 7334442914274893824 to 7334442914224566272. Current Snapshot: MvccSnapshot[applied={T|T < 7334442914274893824 or (T in {7334442914274893824})}]
I20260928 22:44:37.120595 2025 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.120680 2025 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P f3ee90be8c414f4cb7349838e527871b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.125387 2014 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.125483 2014 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P c383b7ca017f42de94e930eabeccd906: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:37.125907 2015 tablet_replica.cc:439] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c383b7ca017f42de94e930eabeccd906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ee90be8c414f4cb7349838e527871b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41575 } } peers { permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 } } peers { permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } }
I20260928 22:44:37.126004 2015 tablet_replica.cc:442] T a89b44f11ab249108c39040146bd4b6d P ac8c885b3a514f85a6bdc5fe55a9a3ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:37.126579 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.123658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38757 (local address 127.0.98.2:51599)
0928 22:44:37.123830 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.123840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.123890 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.124010 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.124010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.124012 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:37.124052 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.124054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.124818 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.124819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.125621 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.125624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.126189 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.126190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.126249 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.126265 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.126442 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.126444 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.126478 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.126524 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260928 22:44:37.126579 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.123726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51599 (local address 127.0.98.3:38757)
0928 22:44:37.123766 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.123768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.123893 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:44:37.123940 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.123940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.123950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.124003 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:37.124088 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.124750 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.125688 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.125937 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.125994 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.126345 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.126379 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.126419 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.126488 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.126510 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:44:37.128525 1661 raft_consensus.cc:1261] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Refusing update from remote peer f3ee90be8c414f4cb7349838e527871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:37.128525 1941 raft_consensus.cc:1261] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Refusing update from remote peer f3ee90be8c414f4cb7349838e527871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:37.128736 2013 consensus_queue.cc:1035] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c383b7ca017f42de94e930eabeccd906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:37.128845 2025 consensus_queue.cc:1035] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8c885b3a514f85a6bdc5fe55a9a3ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:37.136708 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.133882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60335 (local address 127.0.98.2:41575)
0928 22:44:37.133923 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.133924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.134054 (+ 130us) server_negotiation.cc:407] Connection header received
0928 22:44:37.134091 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.134092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.134104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.134173 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:37.134265 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.134991 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.135930 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.136240 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.136307 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.136427 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.136458 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.136518 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.136610 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.136631 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:44:37.136951 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.133842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41575 (local address 127.0.98.3:60335)
0928 22:44:37.133985 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:37.133995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:37.134052 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:37.134173 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:37.134173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:37.134175 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:37.134229 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.134231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.135071 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.135072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.135859 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:37.135861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.136303 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.136304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:37.136349 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.136362 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:37.136543 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:37.136546 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:37.136819 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 22:44:37.136861 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260928 22:44:37.144932 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:37.145056 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:37.145632 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fa9925676e24a44aee6a4b1c01e7af3: Sending DeleteTablet for 3 replicas of tablet 512c9d3f212e42199cc0a264e2d6059b
I20260928 22:44:37.145924 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 512c9d3f212e42199cc0a264e2d6059b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260928 22:44:37.146024 2031 tablet_replica.cc:330] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: stopping tablet replica
I20260928 22:44:37.146066 2031 raft_consensus.cc:2229] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:37.146098 2031 raft_consensus.cc:2258] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:37.146165 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 512c9d3f212e42199cc0a264e2d6059b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59486
I20260928 22:44:37.146258 2030 tablet_replica.cc:330] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: stopping tablet replica
I20260928 22:44:37.146337 2030 raft_consensus.cc:2229] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:37.146392 2030 raft_consensus.cc:2258] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:37.146418 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 512c9d3f212e42199cc0a264e2d6059b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60404
I20260928 22:44:37.146456 2031 ts_tablet_manager.cc:1905] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:37.146497 2032 tablet_replica.cc:330] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: stopping tablet replica
I20260928 22:44:37.146543 2032 raft_consensus.cc:2229] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:37.146574 2032 raft_consensus.cc:2258] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:37.146682 2030 ts_tablet_manager.cc:1905] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:37.146865 2032 ts_tablet_manager.cc:1905] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:37.148241 2032 ts_tablet_manager.cc:1918] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:37.148298 2032 log.cc:1196] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/512c9d3f212e42199cc0a264e2d6059b
I20260928 22:44:37.148285 2031 ts_tablet_manager.cc:1918] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:37.148301 2030 ts_tablet_manager.cc:1918] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:37.148339 2031 log.cc:1196] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/512c9d3f212e42199cc0a264e2d6059b
I20260928 22:44:37.148343 2030 log.cc:1196] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/512c9d3f212e42199cc0a264e2d6059b
I20260928 22:44:37.148494 2032 ts_tablet_manager.cc:1939] T 512c9d3f212e42199cc0a264e2d6059b P c383b7ca017f42de94e930eabeccd906: Deleting consensus metadata
I20260928 22:44:37.148525 2030 ts_tablet_manager.cc:1939] T 512c9d3f212e42199cc0a264e2d6059b P f3ee90be8c414f4cb7349838e527871b: Deleting consensus metadata
I20260928 22:44:37.148543 2031 ts_tablet_manager.cc:1939] T 512c9d3f212e42199cc0a264e2d6059b P ac8c885b3a514f85a6bdc5fe55a9a3ca: Deleting consensus metadata
I20260928 22:44:37.148833 1488 catalog_manager.cc:4841] TS f3ee90be8c414f4cb7349838e527871b (127.0.98.2:41575): tablet 512c9d3f212e42199cc0a264e2d6059b (table test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]) successfully deleted
I20260928 22:44:37.148845 1489 catalog_manager.cc:4841] TS c383b7ca017f42de94e930eabeccd906 (127.0.98.3:38757): tablet 512c9d3f212e42199cc0a264e2d6059b (table test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]) successfully deleted
I20260928 22:44:37.148939 1489 catalog_manager.cc:4841] TS ac8c885b3a514f85a6bdc5fe55a9a3ca (127.0.98.1:33725): tablet 512c9d3f212e42199cc0a264e2d6059b (table test-table [id=6dc0102d6f8c4617afe7fd0c7e179765]) successfully deleted
Sep 28 22:44:37 dist-test-slave-pxr1 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:37.161479 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.146816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.0.98.1:33725)
0928 22:44:37.146851 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.146853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.152911 (+ 6058us) server_negotiation.cc:407] Connection header received
0928 22:44:37.152950 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.152951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.152963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.153034 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:37.153596 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.154374 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.155236 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.155543 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.155616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.158507 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:37.158519 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:37.158519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:37.158526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:37.160165 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.160477 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.160481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.160482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.160548 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.160782 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.160783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.160784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.160969 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:37.161088 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.161311 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.161402 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:44:37.162066 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32822
I20260928 22:44:37.170868 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:37.163606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.62:46485)
0928 22:44:37.163649 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:37.163651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:37.164866 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 22:44:37.164919 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:37.164920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:37.164934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:37.164986 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:37.165452 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.166252 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.166922 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:37.167246 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:37.167313 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:37.168554 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:37.168556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:37.168556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:37.168563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:37.169882 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.170181 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.170183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.170183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.170239 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:37.170398 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:37.170399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:37.170399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:37.170519 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:37.170601 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:37.170700 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:44:37.170776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:44:37.171106 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45216
I20260928 22:44:37.171603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1576
I20260928 22:44:37.177209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1716
I20260928 22:44:37.185640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1856
I20260928 22:44:37.190287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 1472
2026-09-28T22:44:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2318 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:37 dist-test-slave-pxr1 krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:37 dist-test-slave-pxr1 krb5kdc[2070](info): set up 2 sockets
Sep 28 22:44:37 dist-test-slave-pxr1 krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260928 22:44:38.140254 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:38.247236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:38.326069 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:38.326287 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:38.326308 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:38.327441 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:38.327478 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:44:38.327490 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:38.327502 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:38.327518 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:38.327529 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:38.329068 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34191
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:38.329285 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:38.329452 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:38.333984 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.334096 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.334146 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.334206 2083 server_base.cc:1034] running on GCE node
I20260928 22:44:38.334497 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:38.334800 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:38.335940 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790635478335923 us; error 34 us; skew 500 ppm
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:38.338359 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:38.338799 2083 webserver.cc:460] Webserver started at http://127.0.98.62:34579/ using document root <none> and password file <none>
I20260928 22:44:38.338999 2083 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:38.339056 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:38.339178 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:38.340210 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "a27fb123c86c42939733f3193d17280a"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "48f9127b98b6f7557fdd82c17d0f7c27"
server_key_iv: "5aeb2eede1af8216412fa1a8b48549eb"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.340533 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "a27fb123c86c42939733f3193d17280a"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "48f9127b98b6f7557fdd82c17d0f7c27"
server_key_iv: "5aeb2eede1af8216412fa1a8b48549eb"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.341799 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:38.342685 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.342852 2083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:38.342944 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "a27fb123c86c42939733f3193d17280a"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "48f9127b98b6f7557fdd82c17d0f7c27"
server_key_iv: "5aeb2eede1af8216412fa1a8b48549eb"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.343014 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:38.357296 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:38.359089 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:38.359225 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:38.363374 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34191
I20260928 22:44:38.363437 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34191 every 8 connection(s)
I20260928 22:44:38.363737 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:38.364356 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.366657 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Bootstrap starting.
I20260928 22:44:38.367229 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.367491 2161 log.cc:824] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:38.368172 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: No bootstrap required, opened a new log
I20260928 22:44:38.370247 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260928 22:44:38.370389 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.370410 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27fb123c86c42939733f3193d17280a, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.370560 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260928 22:44:38.370626 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:38.370657 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:38.370702 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:38.371198 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260928 22:44:38.371281 2161 leader_election.cc:302] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27fb123c86c42939733f3193d17280a; no voters:
I20260928 22:44:38.371465 2161 leader_election.cc:288] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:38.371495 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:38.371578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2083
I20260928 22:44:38.371600 2164 raft_consensus.cc:686] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [term 1 LEADER]: Becoming Leader. State: Replica: a27fb123c86c42939733f3193d17280a, State: Running, Role: LEADER
I20260928 22:44:38.371661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:38.371670 2164 consensus_queue.cc:237] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } }
I20260928 22:44:38.371713 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:38.371832 392 external_mini_cluster.cc:1428] Setting key 62d33851b29cdd7f55f7a8eb5725560d
I20260928 22:44:38.371984 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a27fb123c86c42939733f3193d17280a. Latest consensus state: current_term: 1 leader_uuid: "a27fb123c86c42939733f3193d17280a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } } }
I20260928 22:44:38.372208 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:38.372143 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a27fb123c86c42939733f3193d17280a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27fb123c86c42939733f3193d17280a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34191 } } }
I20260928 22:44:38.372457 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:38.372629 2174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:38.373157 2174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:38.375715 2174 catalog_manager.cc:1347] Generated new cluster ID: ff3df8de408f47189de0083841ffaf67
I20260928 22:44:38.375770 2174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.382933 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.372423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.98.62:34191)
0928 22:44:38.372588 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.372598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.372895 (+ 297us) server_negotiation.cc:407] Connection header received
0928 22:44:38.373087 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.373095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.373162 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.373306 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.374149 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.375115 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.375976 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.376601 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.376744 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.379368 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.379373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.379373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.379385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.381274 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.381646 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.381649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.381650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.381756 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.381939 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.381939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.381940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.382233 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.382370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.382472 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.382710 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 22:44:38.397814 2174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:38.398343 2174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:38.401013 2174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Generated new TSK 0
I20260928 22:44:38.401203 2174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:38.425769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34191
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:38.504565 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:38.504804 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:38.504833 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:38.505990 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:38.506034 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:38.506057 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:44:38.506068 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:38.506090 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:38.507547 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:38.507753 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:38.507966 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:38.512633 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.512640 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.512652 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.512816 2187 server_base.cc:1034] running on GCE node
I20260928 22:44:38.513022 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:38.513236 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:38.514384 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790635478514366 us; error 35 us; skew 500 ppm
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:38.516649 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:38.517081 2187 webserver.cc:460] Webserver started at http://127.0.98.1:40109/ using document root <none> and password file <none>
I20260928 22:44:38.517283 2187 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:38.517330 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:38.517450 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:38.518460 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "900073c787d640a0a75f5f5c001cd534"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "30cdce93158f274baae76e87afd2b9c0"
server_key_iv: "c26fa14aaa4c439496d7234bfb257a14"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.518797 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "900073c787d640a0a75f5f5c001cd534"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "30cdce93158f274baae76e87afd2b9c0"
server_key_iv: "c26fa14aaa4c439496d7234bfb257a14"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.520052 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:38.520936 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.521112 2187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:38.521202 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "900073c787d640a0a75f5f5c001cd534"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "30cdce93158f274baae76e87afd2b9c0"
server_key_iv: "c26fa14aaa4c439496d7234bfb257a14"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.521276 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:38.533536 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:38.535302 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:38.535434 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:38.535913 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:38.535957 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.535986 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:38.536003 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.542358 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43373
I20260928 22:44:38.542433 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43373 every 8 connection(s)
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.542750 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:38.545368 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.536373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45401 (local address 127.0.98.62:34191)
0928 22:44:38.536422 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.536424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.536674 (+ 250us) server_negotiation.cc:407] Connection header received
0928 22:44:38.537224 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.537225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.537242 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.537293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.538095 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.538856 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.540349 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.540655 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.540708 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.543123 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.543125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.543125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.543136 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.544238 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.544582 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.544583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.544583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.544628 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.544842 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.544843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.544843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.544943 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.544988 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.545242 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.545305 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:44:38.545542 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.536377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.1:45401)
0928 22:44:38.536556 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.536592 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.537124 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.537366 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.537370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.537732 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.538010 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.538016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.538952 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.538954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.540258 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.540261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.540757 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.540759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.540911 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.541485 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.541495 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.543009 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.544302 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.544307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.544309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.544525 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.544668 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.544669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.544669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.544795 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.544986 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.544989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.545137 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.545214 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.545335 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260928 22:44:38.545984 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34191
I20260928 22:44:38.546070 2318 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:38.546275 2318 heartbeater.cc:511] Master 127.0.98.62:34191 requested a full tablet report, sending...
I20260928 22:44:38.546787 2114 ts_manager.cc:194] Registered new tserver with Master: 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373)
I20260928 22:44:38.547423 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45401
I20260928 22:44:38.550635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2187
I20260928 22:44:38.550722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:38.550881 392 external_mini_cluster.cc:1428] Setting key 1ae7e4b93fa50d6180cd44ad85f893ea
I20260928 22:44:38.551877 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.546954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.1:40939)
0928 22:44:38.547082 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.547091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.547150 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.547265 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.547265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.547420 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.547466 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.547468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.548300 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.548301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.548839 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.548841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.549160 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.549161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.549236 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.549578 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.549582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.549829 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.550994 (+ 1165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.550995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.550995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.551183 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.551349 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.551349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.551350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.551408 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.551619 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.551620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.551663 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.551729 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.551805 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260928 22:44:38.551913 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.546994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40939 (local address 127.0.98.62:34191)
0928 22:44:38.547053 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.547055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.547128 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:44:38.547200 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.547201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.547211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.547251 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.547536 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.548221 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.548911 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.549105 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.549156 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.549903 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.549904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.549904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.549911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.550915 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.551254 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.551255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.551255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.551302 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.551455 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.551456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.551456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.551562 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.551619 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.551762 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.551837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:38.573756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34191
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:38.651643 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:38.651840 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:38.651873 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:38.653088 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:38.653131 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:38.653158 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:44:38.653172 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:38.653195 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:38.654620 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:38.654846 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:38.655053 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:38.659750 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.659799 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.659823 2327 server_base.cc:1034] running on GCE node
W20260928 22:44:38.659765 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.660140 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:38.660364 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:38.661495 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790635478661473 us; error 33 us; skew 500 ppm
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:38.663843 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:38.664264 2327 webserver.cc:460] Webserver started at http://127.0.98.2:46737/ using document root <none> and password file <none>
I20260928 22:44:38.664469 2327 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:38.664515 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:38.664645 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:38.665694 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "a419ade95f5446b8b7604b0e64e772c2"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "f00db9ef0f4bd1967448677ed7be2647"
server_key_iv: "6c664db558309119680cf620d52b7ead"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.666015 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a419ade95f5446b8b7604b0e64e772c2"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "f00db9ef0f4bd1967448677ed7be2647"
server_key_iv: "6c664db558309119680cf620d52b7ead"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.667227 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:38.667896 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.668036 2327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:38.668109 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "a419ade95f5446b8b7604b0e64e772c2"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "f00db9ef0f4bd1967448677ed7be2647"
server_key_iv: "6c664db558309119680cf620d52b7ead"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.668169 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:38.683841 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:38.685990 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:38.686128 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:38.686661 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:38.686700 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.686729 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:38.686769 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.693244 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41773
I20260928 22:44:38.693317 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41773 every 8 connection(s)
I20260928 22:44:38.693598 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:38.695724 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.687046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47955 (local address 127.0.98.62:34191)
0928 22:44:38.687101 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.687104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.687320 (+ 216us) server_negotiation.cc:407] Connection header received
0928 22:44:38.687833 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.687835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.687848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.687923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.688529 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.689128 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.690104 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.690301 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.690362 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.693187 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.693188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.693189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.693196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.694547 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.694890 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.694892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.694892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.694947 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.695148 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.695149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.695149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.695259 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.695311 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.695563 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.695636 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:44:38.695884 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.687031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.2:47955)
0928 22:44:38.687190 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.687231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.687738 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.687994 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.687997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.688246 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.688437 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.688441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.689203 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.689204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.690023 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.690025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.690364 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.690365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.690500 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.690985 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.690997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.693088 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.694617 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.694622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.694624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.694824 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.694998 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.694999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.694999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.695096 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.695309 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.695314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.695449 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.695535 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.695682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260928 22:44:38.696302 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34191
I20260928 22:44:38.696383 2458 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:38.696547 2458 heartbeater.cc:511] Master 127.0.98.62:34191 requested a full tablet report, sending...
I20260928 22:44:38.696933 2115 ts_manager.cc:194] Registered new tserver with Master: a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:38.697408 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47955
I20260928 22:44:38.697885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2327
I20260928 22:44:38.697975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:38.698154 392 external_mini_cluster.cc:1428] Setting key da2793c52561fbbc5e624d54fd940c6d
I20260928 22:44:38.702683 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.697281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56415 (local address 127.0.98.62:34191)
0928 22:44:38.697320 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.697322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.697373 (+ 51us) server_negotiation.cc:407] Connection header received
0928 22:44:38.697446 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.697446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.697453 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.697506 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.697824 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.698481 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.699393 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.699571 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.699620 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.700428 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.700430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.700430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.700436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.701675 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.702014 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.702015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.702015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.702058 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.702234 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.702234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.702235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.702372 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.702421 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.702559 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.702612 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:44:38.702744 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.697204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.2:56415)
0928 22:44:38.697318 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.697328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.697393 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.697511 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.697511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.697680 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.697735 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.697737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.698586 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.698587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.699322 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.699325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.699633 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.699634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.699701 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.700074 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.700079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.700351 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.701751 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.701752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.701753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.701952 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.702120 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.702121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.702121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.702184 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.702429 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.702429 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.702473 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.702544 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.702640 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:38.721541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34191
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:38.799116 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:38.799332 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:38.799366 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:38.800571 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:38.800624 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:38.800652 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:44:38.800666 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:44:38.800689 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:38.802227 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:38.802482 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:38.802712 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:38.807762 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:38.807775 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.807946 2467 server_base.cc:1034] running on GCE node
W20260928 22:44:38.807788 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:38.808214 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:38.808462 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:38.809640 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790635478809602 us; error 53 us; skew 500 ppm
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:38.812206 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:38.812676 2467 webserver.cc:460] Webserver started at http://127.0.98.3:43217/ using document root <none> and password file <none>
I20260928 22:44:38.812994 2467 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:38.813057 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:38.813184 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:38.814232 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "e487968695d24e7fb8be26034ad5e346"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "58ecccfc4c611eab1239e37b82bce165"
server_key_iv: "1d01669b1f4f8335309405add9c597bd"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.814572 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e487968695d24e7fb8be26034ad5e346"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "58ecccfc4c611eab1239e37b82bce165"
server_key_iv: "1d01669b1f4f8335309405add9c597bd"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.815907 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:38.816699 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.816956 2467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:38.817060 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "e487968695d24e7fb8be26034ad5e346"
format_stamp: "Formatted at 2026-09-28 22:44:38 on dist-test-slave-pxr1"
server_key: "58ecccfc4c611eab1239e37b82bce165"
server_key_iv: "1d01669b1f4f8335309405add9c597bd"
server_key_version: "encryptionkey@0"
I20260928 22:44:38.817122 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:38.831700 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:38.833822 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:38.833968 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:38.834492 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:38.834532 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.834560 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:38.834578 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.841018 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39207
I20260928 22:44:38.841109 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39207 every 8 connection(s)
I20260928 22:44:38.841472 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:38.843550 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.834878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50497 (local address 127.0.98.62:34191)
0928 22:44:38.834927 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.834929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.835148 (+ 219us) server_negotiation.cc:407] Connection header received
0928 22:44:38.835680 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.835682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.835695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.835746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.836432 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.837026 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.838035 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.838291 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.838350 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.840904 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.840906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.840906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.840914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.842372 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.842745 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.842746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.842746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.842797 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.843069 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.843069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.843070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.843169 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.843214 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.843429 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.843489 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:44:38.843726 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.834876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.3:50497)
0928 22:44:38.835027 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.835063 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.835580 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.835819 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.835823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.836131 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.836336 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.836341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.837115 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.837116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.837945 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.837948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.838363 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.838363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.838512 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.839013 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.839021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.840813 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.842445 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.842449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.842450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.842684 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.842903 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.842904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.842905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.843012 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.843214 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.843216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.843328 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.843412 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.843535 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260928 22:44:38.844231 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34191
I20260928 22:44:38.844310 2598 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:38.844491 2598 heartbeater.cc:511] Master 127.0.98.62:34191 requested a full tablet report, sending...
I20260928 22:44:38.844866 2114 ts_manager.cc:194] Registered new tserver with Master: e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207)
I20260928 22:44:38.845269 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50497
I20260928 22:44:38.845780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2467
I20260928 22:44:38.845870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:38.846043 392 external_mini_cluster.cc:1428] Setting key 72c6e6d6664b34813813c951a896cb4f
I20260928 22:44:38.846889 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:44:38.851042 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.845196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.98.3:39291)
0928 22:44:38.845325 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.845334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.845391 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.845523 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.845524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.845711 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.845767 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.845769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.846620 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.846621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.847288 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.847290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.847779 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.847780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.847847 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.848275 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.848279 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.848599 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.850034 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.850035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.850035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.850219 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.850432 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.850432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.850433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.850479 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.850758 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.850759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.850804 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.850870 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.850964 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 22:44:38.851065 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.845267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39291 (local address 127.0.98.62:34191)
0928 22:44:38.845382 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.845383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.845393 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:38.845435 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.845436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.845446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.845509 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.845845 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.846518 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.847355 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.847686 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.847765 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.848692 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.848694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.848694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.848701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.849952 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.850302 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.850303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.850304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.850372 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.850528 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.850529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.850529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.850646 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.850704 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.850903 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.850980 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.870441 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.863503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.0.98.62:34191)
0928 22:44:38.863563 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.863565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.863573 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:38.863614 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.863615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.863624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.863671 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.864291 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.864898 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.865704 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.865911 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.865970 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.868254 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.868255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.868256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.868262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.869574 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.869903 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.869904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.869905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.869949 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.870082 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.870083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.870083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.870177 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.870243 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.870317 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.870373 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:44:38.872291 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:38.874064 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:38.880441 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.0.1:52226)
0928 22:44:38.876908 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.876916 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.876964 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.877158 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.877159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.877161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:38.877215 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.877217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878160 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.878160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.879411 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.879414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.880114 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.880115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.880198 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.880212 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.880282 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.880359 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":73,"threads_started":1}
I20260928 22:44:38.880458 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.98.3:39207)
0928 22:44:38.876523 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.876528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.877002 (+ 474us) server_negotiation.cc:407] Connection header received
0928 22:44:38.877046 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.877048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.877064 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.877154 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:38.877307 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878082 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.879486 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.880029 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.880086 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.880129 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.880307 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.880348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 22:44:38.880458 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.98.2:41773)
0928 22:44:38.876797 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.876800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.877114 (+ 314us) server_negotiation.cc:407] Connection header received
0928 22:44:38.877282 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.877285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.877302 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.877381 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:38.877545 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878554 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.879559 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.880101 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.880173 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.880230 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.880334 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.880372 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260928 22:44:38.880681 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41773 (local address 127.0.0.1:48090)
0928 22:44:38.877023 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.877030 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.877067 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.877447 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.877448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.877450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:38.877484 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.877485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878623 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.878624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.879478 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.879481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.880185 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.880187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.880238 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.880242 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.880287 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.880633 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":300,"thread_start_us":28,"threads_started":1}
I20260928 22:44:38.881336 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43373 (local address 127.0.0.1:58496)
0928 22:44:38.876649 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.876680 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.876801 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.877021 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.877025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.877028 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:38.877104 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.877106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878396 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.878397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.879499 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.879501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.881097 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.881098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.881172 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.881176 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.881234 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.881273 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":8,"thread_start_us":79,"threads_started":1}
I20260928 22:44:38.881384 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.876388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.0.98.1:43373)
0928 22:44:38.876563 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.876568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.876760 (+ 192us) server_negotiation.cc:407] Connection header received
0928 22:44:38.876898 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.876901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.876918 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.876998 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:38.877189 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.878306 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.880272 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.881024 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.881097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.881158 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.881250 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.881287 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260928 22:44:38.881659 2532 tablet_service.cc:1467] Processing CreateTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (DEFAULT_TABLE table=test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:38.881683 2392 tablet_service.cc:1467] Processing CreateTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (DEFAULT_TABLE table=test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:38.881979 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb78c8ba1c6d49a48d2d3e3b7e544999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.882031 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb78c8ba1c6d49a48d2d3e3b7e544999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.882033 2252 tablet_service.cc:1467] Processing CreateTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (DEFAULT_TABLE table=test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:38.882320 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb78c8ba1c6d49a48d2d3e3b7e544999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.884802 2618 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Bootstrap starting.
I20260928 22:44:38.886096 2618 tablet_bootstrap.cc:654] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.886108 2619 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Bootstrap starting.
I20260928 22:44:38.886511 2618 log.cc:824] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:38.886670 2619 tablet_bootstrap.cc:654] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.887091 2619 log.cc:824] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:38.887398 2618 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: No bootstrap required, opened a new log
I20260928 22:44:38.887418 2620 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Bootstrap starting.
I20260928 22:44:38.887472 2618 ts_tablet_manager.cc:1397] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:44:38.887789 2619 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: No bootstrap required, opened a new log
I20260928 22:44:38.887949 2619 ts_tablet_manager.cc:1397] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:38.889434 2619 raft_consensus.cc:348] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.889428 2618 raft_consensus.cc:348] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.889572 2618 raft_consensus.cc:374] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.889573 2619 raft_consensus.cc:374] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.889593 2618 raft_consensus.cc:729] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.889605 2619 raft_consensus.cc:729] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.889708 2618 consensus_queue.cc:260] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.890017 2618 ts_tablet_manager.cc:1428] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:44:38.890064 2598 heartbeater.cc:503] Master 127.0.98.62:34191 was elected leader, sending a full tablet report...
I20260928 22:44:38.890348 2620 tablet_bootstrap.cc:654] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.890640 2620 log.cc:824] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:38.890900 2619 consensus_queue.cc:260] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.891214 2619 ts_tablet_manager.cc:1428] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:44:38.891244 2458 heartbeater.cc:503] Master 127.0.98.62:34191 was elected leader, sending a full tablet report...
I20260928 22:44:38.891587 2620 tablet_bootstrap.cc:492] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: No bootstrap required, opened a new log
I20260928 22:44:38.891666 2620 ts_tablet_manager.cc:1397] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:44:38.893213 2620 raft_consensus.cc:348] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.893349 2620 raft_consensus.cc:374] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.893373 2620 raft_consensus.cc:729] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.893484 2620 consensus_queue.cc:260] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.893719 2620 ts_tablet_manager.cc:1428] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:44:38.893777 2318 heartbeater.cc:503] Master 127.0.98.62:34191 was elected leader, sending a full tablet report...
I20260928 22:44:38.900008 2623 raft_consensus.cc:482] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:38.900113 2623 raft_consensus.cc:504] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.900391 2623 leader_election.cc:288] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:38.903254 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.900838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51967 (local address 127.0.98.2:41773)
0928 22:44:38.900886 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.900888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.900898 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:38.900942 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.900943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.900953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.900997 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:38.901129 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.901706 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.902604 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.902961 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.903017 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.903028 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.903154 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.903187 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:44:38.903260 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.900629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40309 (local address 127.0.98.1:43373)
0928 22:44:38.900684 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.900686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.900696 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:38.900821 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.900822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.900835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.900884 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:38.901044 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.901689 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.902599 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.902970 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.903030 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.903039 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.903164 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.903197 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:38.903275 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.900560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43373 (local address 127.0.98.3:40309)
0928 22:44:38.900609 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.900622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.900675 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.900920 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.900921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.900923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:38.900986 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.900987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.901760 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.901761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.902517 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.902518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.903036 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.903037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.903107 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.903110 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.903153 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.903193 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:44:38.903630 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb78c8ba1c6d49a48d2d3e3b7e544999" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534" is_pre_election: true
I20260928 22:44:38.903282 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.900660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41773 (local address 127.0.98.3:51967)
0928 22:44:38.900825 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.900835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.900888 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.900999 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.901000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.901001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:38.901073 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.901074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.901780 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.901781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.902526 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.902528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.903026 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.903027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.903097 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.903101 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.903144 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.903183 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260928 22:44:38.903769 2272 raft_consensus.cc:2454] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:38.903965 2486 leader_election.cc:302] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:38.903982 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb78c8ba1c6d49a48d2d3e3b7e544999" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2" is_pre_election: true
I20260928 22:44:38.904075 2623 raft_consensus.cc:2781] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:38.904130 2623 raft_consensus.cc:482] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:38.904121 2412 raft_consensus.cc:2454] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:38.904145 2623 raft_consensus.cc:3037] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:38.904865 2623 raft_consensus.cc:504] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.904971 2623 leader_election.cc:288] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Requested vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:38.905121 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb78c8ba1c6d49a48d2d3e3b7e544999" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2"
I20260928 22:44:38.905128 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb78c8ba1c6d49a48d2d3e3b7e544999" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534"
I20260928 22:44:38.905193 2272 raft_consensus.cc:3037] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:38.905189 2412 raft_consensus.cc:3037] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:38.905843 2412 raft_consensus.cc:2454] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:38.905848 2272 raft_consensus.cc:2454] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:38.905975 2486 leader_election.cc:302] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a419ade95f5446b8b7604b0e64e772c2, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:38.906056 2623 raft_consensus.cc:2781] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:38.906188 2623 raft_consensus.cc:686] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 1 LEADER]: Becoming Leader. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Running, Role: LEADER
I20260928 22:44:38.906263 2623 consensus_queue.cc:237] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } }
I20260928 22:44:38.907065 2114 catalog_manager.cc:5495] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 reported cstate change: term changed from 0 to 1, leader changed from <none> to e487968695d24e7fb8be26034ad5e346 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } health_report { overall_health: UNKNOWN } } }
W20260928 22:44:38.944341 2459 tablet.cc:2369] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:44:38 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:38.947857 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.941486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.0.98.62:34191)
0928 22:44:38.941543 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:38.941545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:38.941648 (+ 103us) server_negotiation.cc:407] Connection header received
0928 22:44:38.941692 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:38.941693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:38.941703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:38.941751 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:38.942158 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.942646 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.943289 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.943467 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.943512 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.945603 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:38.945604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:38.945604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:38.945611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:38.946983 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.947307 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.947307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.947308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.947353 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:38.947480 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:38.947481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:38.947481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:38.947583 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:38.947626 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:38.947726 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:44:38.947788 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:44:38.947942 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:38.941438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34191 (local address 127.0.0.1:36054)
0928 22:44:38.941574 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:38.941583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:38.941651 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:38.941766 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:38.941766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:38.942054 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:38.942103 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.942106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.942701 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.942702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.943232 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:38.943233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:38.943522 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:38.943523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:38.943561 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:38.944075 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:38.944079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:38.945502 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:38.947059 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.947066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.947068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.947255 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.947395 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:38.947396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:38.947396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:38.947443 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:38.947622 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:38.947626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:38.947665 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 22:44:38.947706 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 22:44:38.947834 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":44,"thread_start_us":90,"threads_started":1}
I20260928 22:44:38.949632 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36054:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:44:38.949820 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:38.951195 2252 tablet_service.cc:1467] Processing CreateTablet for tablet de446e44d1ca4805816a6f56c501cf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b5f7bd64a19442c8c2c05a07872dbaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:38.951221 2532 tablet_service.cc:1467] Processing CreateTablet for tablet de446e44d1ca4805816a6f56c501cf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b5f7bd64a19442c8c2c05a07872dbaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:38.951349 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de446e44d1ca4805816a6f56c501cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.951350 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de446e44d1ca4805816a6f56c501cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.952423 2620 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: Bootstrap starting.
I20260928 22:44:38.952533 2392 tablet_service.cc:1467] Processing CreateTablet for tablet de446e44d1ca4805816a6f56c501cf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b5f7bd64a19442c8c2c05a07872dbaa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:44:38.952695 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de446e44d1ca4805816a6f56c501cf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:38.953092 2618 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: Bootstrap starting.
I20260928 22:44:38.953131 2620 tablet_bootstrap.cc:654] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.953469 2618 tablet_bootstrap.cc:654] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.953792 2620 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: No bootstrap required, opened a new log
I20260928 22:44:38.953832 2620 ts_tablet_manager.cc:1397] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:38.953907 2619 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: Bootstrap starting.
I20260928 22:44:38.953967 2620 raft_consensus.cc:348] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.954036 2618 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: No bootstrap required, opened a new log
I20260928 22:44:38.954047 2620 raft_consensus.cc:374] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.954066 2620 raft_consensus.cc:729] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.954066 2618 ts_tablet_manager.cc:1397] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:38.954116 2620 consensus_queue.cc:260] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.954207 2620 ts_tablet_manager.cc:1428] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.954190 2618 raft_consensus.cc:348] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.954254 2618 raft_consensus.cc:374] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.954272 2618 raft_consensus.cc:729] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.954272 2626 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:38.954315 2618 consensus_queue.cc:260] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.954348 2626 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:38.954402 2618 ts_tablet_manager.cc:1428] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.954464 2619 tablet_bootstrap.cc:654] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:38.954501 2623 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:38.954555 2623 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:38.955204 2619 tablet_bootstrap.cc:492] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: No bootstrap required, opened a new log
I20260928 22:44:38.955247 2619 ts_tablet_manager.cc:1397] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:38.955372 2619 raft_consensus.cc:348] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.955451 2619 raft_consensus.cc:374] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:38.955472 2619 raft_consensus.cc:729] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Initialized, Role: FOLLOWER
I20260928 22:44:38.955515 2619 consensus_queue.cc:260] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:38.955601 2619 ts_tablet_manager.cc:1428] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:38.955654 2625 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:38.955722 2625 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 22:44:39.043497 2319 tablet.cc:2369] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:44:39.092064 2599 tablet.cc:2369] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:39.271164 2623 raft_consensus.cc:482] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:39.271268 2623 raft_consensus.cc:504] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.271396 2623 leader_election.cc:288] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:39.271679 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de446e44d1ca4805816a6f56c501cf65" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534" is_pre_election: true
I20260928 22:44:39.271703 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de446e44d1ca4805816a6f56c501cf65" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2" is_pre_election: true
I20260928 22:44:39.271780 2272 raft_consensus.cc:2454] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:39.271823 2412 raft_consensus.cc:2454] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:39.271943 2486 leader_election.cc:302] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:39.272034 2623 raft_consensus.cc:2781] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:39.272075 2623 raft_consensus.cc:482] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:39.272091 2623 raft_consensus.cc:3037] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.272835 2623 raft_consensus.cc:504] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.272941 2623 leader_election.cc:288] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Requested vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:39.273123 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de446e44d1ca4805816a6f56c501cf65" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534"
I20260928 22:44:39.273146 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de446e44d1ca4805816a6f56c501cf65" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2"
I20260928 22:44:39.273207 2272 raft_consensus.cc:3037] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.273206 2412 raft_consensus.cc:3037] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.273797 2272 raft_consensus.cc:2454] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:39.273801 2412 raft_consensus.cc:2454] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:39.273962 2486 leader_election.cc:302] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:39.274072 2623 raft_consensus.cc:2781] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:39.274135 2623 raft_consensus.cc:686] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [term 1 LEADER]: Becoming Leader. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Running, Role: LEADER
I20260928 22:44:39.274189 2623 consensus_queue.cc:237] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.274499 2628 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: TxnStatusTablet state changed. Reason: New leader e487968695d24e7fb8be26034ad5e346. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.274581 2628 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:39.274827 2640 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:44:39.274803 2115 catalog_manager.cc:5495] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 reported cstate change: term changed from 0 to 1, leader changed from <none> to e487968695d24e7fb8be26034ad5e346 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } health_report { overall_health: HEALTHY } } }
I20260928 22:44:39.274963 2640 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:44:39.308548 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.305846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.98.3:39207)
0928 22:44:39.305892 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.305894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.305930 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:44:39.306030 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.306030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.306047 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.306104 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.306236 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.306701 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.307817 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.308013 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.308068 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.308249 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.308346 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.308406 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.308451 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.308469 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:44:39.308583 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.305782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.0.1:52228)
0928 22:44:39.305867 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.305880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.305951 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.306125 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.306127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.306128 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:39.306181 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.306184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.306772 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.306773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.307720 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.307722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.308084 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.308085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.308149 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.308168 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.308401 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.308407 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.308450 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.308497 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:44:39.310469 2412 raft_consensus.cc:1261] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.310469 2272 raft_consensus.cc:1261] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.310814 2623 consensus_queue.cc:1035] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.310953 2628 consensus_queue.cc:1035] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.312041 2626 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: TxnStatusTablet state changed. Reason: New leader e487968695d24e7fb8be26034ad5e346. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.312132 2626 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:39.312170 2625 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: TxnStatusTablet state changed. Reason: New leader e487968695d24e7fb8be26034ad5e346. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.312261 2625 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:39.313345 2628 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.313436 2628 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:39.313480 2628 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.313514 2628 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P e487968695d24e7fb8be26034ad5e346: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:44:39.313973 2625 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.314062 2625 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P a419ade95f5446b8b7604b0e64e772c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:39.314047 2626 tablet_replica.cc:439] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } } }
I20260928 22:44:39.314131 2626 tablet_replica.cc:442] T de446e44d1ca4805816a6f56c501cf65 P 900073c787d640a0a75f5f5c001cd534: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:44:39.314354 2645 mvcc.cc:205] Tried to move back new op lower bound from 7334442923253194752 to 7334442923107561472. Current Snapshot: MvccSnapshot[applied={T|T < 7334442923253194752}]
I20260928 22:44:39.314822 2643 mvcc.cc:205] Tried to move back new op lower bound from 7334442923253194752 to 7334442923107561472. Current Snapshot: MvccSnapshot[applied={T|T < 7334442923253194752 or (T in {7334442923253194752})}]
I20260928 22:44:39.318856 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.316407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.98.3:39207)
0928 22:44:39.316443 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.316444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.316451 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:39.316468 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.316469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.316476 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.316518 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.316649 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.317227 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.318325 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.318509 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.318552 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.318691 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.318714 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.318743 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.318792 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.318807 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:44:39.323076 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.321156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43519 (local address 127.0.98.3:39207)
0928 22:44:39.321197 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.321199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.321204 (+ 5us) server_negotiation.cc:407] Connection header received
0928 22:44:39.321218 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.321218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.321224 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.321264 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.321371 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.321787 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.322476 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.322641 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.322690 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.322845 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.322874 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.322921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.322962 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.322984 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":26}
I20260928 22:44:39.323077 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.321001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.98.3:43519)
0928 22:44:39.321065 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.321074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.321110 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.321271 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.321272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.321273 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:39.321320 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.321321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.321850 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.321851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.322409 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.322410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.322693 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.322694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.322754 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.322770 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.322921 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.322922 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.322961 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.323001 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260928 22:44:39.325515 2272 raft_consensus.cc:1261] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.325515 2412 raft_consensus.cc:1261] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.325697 2649 consensus_queue.cc:1035] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.325809 2623 consensus_queue.cc:1035] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.328058 2644 mvcc.cc:205] Tried to move back new op lower bound from 7334442923316113408 to 7334442921600770048. Current Snapshot: MvccSnapshot[applied={T|T < 7334442923316113408}]
I20260928 22:44:39.336503 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:39.336607 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:39.337363 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Sending DeleteTablet for 3 replicas of tablet bb78c8ba1c6d49a48d2d3e3b7e544999
I20260928 22:44:39.337662 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260928 22:44:39.337688 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260928 22:44:39.337832 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet bb78c8ba1c6d49a48d2d3e3b7e544999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52226
I20260928 22:44:39.338060 2659 tablet_replica.cc:330] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: stopping tablet replica
I20260928 22:44:39.338060 2658 tablet_replica.cc:330] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: stopping tablet replica
I20260928 22:44:39.338030 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:39.338171 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:39.338172 2658 raft_consensus.cc:2229] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:39.338208 2659 raft_consensus.cc:2229] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:39.338230 2658 raft_consensus.cc:2258] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:39.338265 2659 raft_consensus.cc:2258] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:39.338804 2659 ts_tablet_manager.cc:1905] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:39.339401 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 (DEFAULT_TABLE table=test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:39.339447 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 (DEFAULT_TABLE table=test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:39.339525 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3ef1b32d354f18a48cb9f83828d896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:39.339504 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 (DEFAULT_TABLE table=test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:39.339545 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3ef1b32d354f18a48cb9f83828d896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:39.339600 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3ef1b32d354f18a48cb9f83828d896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:39.340622 2619 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Bootstrap starting.
I20260928 22:44:39.340842 2620 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Bootstrap starting.
I20260928 22:44:39.340941 2618 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Bootstrap starting.
I20260928 22:44:39.341041 2619 tablet_bootstrap.cc:654] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:39.341248 2620 tablet_bootstrap.cc:654] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:39.341315 2618 tablet_bootstrap.cc:654] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:39.341658 2619 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: No bootstrap required, opened a new log
I20260928 22:44:39.341701 2619 ts_tablet_manager.cc:1397] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:39.341837 2619 raft_consensus.cc:348] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.341898 2619 raft_consensus.cc:374] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:39.341881 2659 ts_tablet_manager.cc:1918] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:39.341917 2619 raft_consensus.cc:729] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Initialized, Role: FOLLOWER
I20260928 22:44:39.341948 2659 log.cc:1196] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/bb78c8ba1c6d49a48d2d3e3b7e544999
I20260928 22:44:39.341958 2619 consensus_queue.cc:260] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.341974 2618 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: No bootstrap required, opened a new log
I20260928 22:44:39.342015 2618 ts_tablet_manager.cc:1397] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:39.342041 2619 ts_tablet_manager.cc:1428] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:39.342132 2618 raft_consensus.cc:348] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.342195 2659 ts_tablet_manager.cc:1939] T bb78c8ba1c6d49a48d2d3e3b7e544999 P a419ade95f5446b8b7604b0e64e772c2: Deleting consensus metadata
I20260928 22:44:39.342207 2618 raft_consensus.cc:374] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:39.342228 2618 raft_consensus.cc:729] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Initialized, Role: FOLLOWER
I20260928 22:44:39.342270 2618 consensus_queue.cc:260] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.342360 2618 ts_tablet_manager.cc:1428] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:39.342739 2620 tablet_bootstrap.cc:492] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: No bootstrap required, opened a new log
I20260928 22:44:39.342790 2620 ts_tablet_manager.cc:1397] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 22:44:39.342770 2102 catalog_manager.cc:4841] TS a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773): tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (table test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]) successfully deleted
I20260928 22:44:39.342939 2620 raft_consensus.cc:348] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.342995 2658 ts_tablet_manager.cc:1905] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:39.343007 2620 raft_consensus.cc:374] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:39.343138 2620 raft_consensus.cc:729] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Initialized, Role: FOLLOWER
I20260928 22:44:39.343185 2620 consensus_queue.cc:260] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.343277 2620 ts_tablet_manager.cc:1428] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:39.344686 2658 ts_tablet_manager.cc:1918] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:39.344794 2658 log.cc:1196] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/bb78c8ba1c6d49a48d2d3e3b7e544999
I20260928 22:44:39.345081 2658 ts_tablet_manager.cc:1939] T bb78c8ba1c6d49a48d2d3e3b7e544999 P 900073c787d640a0a75f5f5c001cd534: Deleting consensus metadata
I20260928 22:44:39.345471 2102 catalog_manager.cc:4841] TS 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373): tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (table test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]) successfully deleted
I20260928 22:44:39.348856 2660 tablet_replica.cc:330] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: stopping tablet replica
I20260928 22:44:39.348992 2660 raft_consensus.cc:2229] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:39.349141 2660 raft_consensus.cc:2258] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:39.349567 2660 ts_tablet_manager.cc:1905] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:39.351202 2660 ts_tablet_manager.cc:1918] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:39.351276 2660 log.cc:1196] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/bb78c8ba1c6d49a48d2d3e3b7e544999
I20260928 22:44:39.351541 2660 ts_tablet_manager.cc:1939] T bb78c8ba1c6d49a48d2d3e3b7e544999 P e487968695d24e7fb8be26034ad5e346: Deleting consensus metadata
I20260928 22:44:39.351964 2101 catalog_manager.cc:4841] TS e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207): tablet bb78c8ba1c6d49a48d2d3e3b7e544999 (table test-table [id=7ee2a72cb0df4cecb18ccd9708cba5a6]) successfully deleted
I20260928 22:44:39.461195 2625 raft_consensus.cc:482] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:39.461311 2625 raft_consensus.cc:504] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.461678 2625 leader_election.cc:288] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207)
I20260928 22:44:39.465236 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.461902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33653 (local address 127.0.98.1:43373)
0928 22:44:39.462045 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.462047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.462163 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:44:39.462248 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.462249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.462260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.462320 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:39.462550 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.463257 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.464440 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.464887 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.464948 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.464964 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.465105 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.465158 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":53,"threads_started":1}
I20260928 22:44:39.465255 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.462001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44755 (local address 127.0.98.3:39207)
0928 22:44:39.462055 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.462057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.462137 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:44:39.462234 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.462235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.462244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.462301 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:39.462471 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.462961 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.464336 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.464909 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.464979 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.464993 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.465148 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.465193 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:39.465263 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.461837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43373 (local address 127.0.98.2:33653)
0928 22:44:39.462106 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.462116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.462179 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.462362 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.462362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.462364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:39.462424 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.462426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.463342 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.463343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.464367 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.464369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.464964 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.464964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.465027 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.465030 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.465083 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.465128 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260928 22:44:39.465487 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.461837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.98.2:44755)
0928 22:44:39.462031 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.462044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.462171 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.462314 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.462315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.462317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:39.462400 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.462402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.463041 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.463042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.464237 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.464240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.465006 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.465007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.465067 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.465071 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.465365 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.465408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 22:44:39.465701 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3ef1b32d354f18a48cb9f83828d896" candidate_uuid: "a419ade95f5446b8b7604b0e64e772c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e487968695d24e7fb8be26034ad5e346" is_pre_election: true
I20260928 22:44:39.465850 2552 raft_consensus.cc:2454] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a419ade95f5446b8b7604b0e64e772c2 in term 0.
I20260928 22:44:39.465867 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3ef1b32d354f18a48cb9f83828d896" candidate_uuid: "a419ade95f5446b8b7604b0e64e772c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534" is_pre_election: true
I20260928 22:44:39.465965 2272 raft_consensus.cc:2454] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a419ade95f5446b8b7604b0e64e772c2 in term 0.
I20260928 22:44:39.466050 2345 leader_election.cc:302] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a419ade95f5446b8b7604b0e64e772c2, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:39.466197 2625 raft_consensus.cc:2781] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:39.466249 2625 raft_consensus.cc:482] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:39.466279 2625 raft_consensus.cc:3037] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.467216 2625 raft_consensus.cc:504] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.467366 2625 leader_election.cc:288] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [CANDIDATE]: Term 1 election: Requested vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207)
I20260928 22:44:39.467495 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3ef1b32d354f18a48cb9f83828d896" candidate_uuid: "a419ade95f5446b8b7604b0e64e772c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534"
I20260928 22:44:39.467492 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3ef1b32d354f18a48cb9f83828d896" candidate_uuid: "a419ade95f5446b8b7604b0e64e772c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e487968695d24e7fb8be26034ad5e346"
I20260928 22:44:39.467568 2552 raft_consensus.cc:3037] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.467568 2272 raft_consensus.cc:3037] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:39.468294 2272 raft_consensus.cc:2454] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a419ade95f5446b8b7604b0e64e772c2 in term 1.
I20260928 22:44:39.468294 2552 raft_consensus.cc:2454] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a419ade95f5446b8b7604b0e64e772c2 in term 1.
I20260928 22:44:39.468473 2345 leader_election.cc:302] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a419ade95f5446b8b7604b0e64e772c2, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:39.468557 2625 raft_consensus.cc:2781] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:39.468652 2625 raft_consensus.cc:686] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 1 LEADER]: Becoming Leader. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Running, Role: LEADER
I20260928 22:44:39.468745 2625 consensus_queue.cc:237] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:39.469520 2114 catalog_manager.cc:5495] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a419ade95f5446b8b7604b0e64e772c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:39.490105 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.487533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.98.2:41773)
0928 22:44:39.487670 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.487672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.487680 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:39.487713 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.487714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.487733 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.487807 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.487907 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.488385 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.489374 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.489578 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.489622 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.489769 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.489873 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.489923 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.489985 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.490001 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 22:44:39.494985 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.492505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46509 (local address 127.0.98.3:39207)
0928 22:44:39.492547 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.492549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.492664 (+ 115us) server_negotiation.cc:407] Connection header received
0928 22:44:39.492752 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.492753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.492763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.492815 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.492936 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.493374 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.494396 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.494568 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.494607 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.494807 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.494830 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.494860 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.494915 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.494929 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:44:39.495056 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.492439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.98.2:46509)
0928 22:44:39.492602 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.492613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.492672 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.492820 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.492821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.492823 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:39.492882 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.492884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.493451 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.493452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.494317 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.494320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.494649 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.494650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.494715 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.494733 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.494864 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.494866 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.494914 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.494958 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 22:44:39.497114 2272 raft_consensus.cc:1261] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Refusing update from remote peer a419ade95f5446b8b7604b0e64e772c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.497249 2552 raft_consensus.cc:1261] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Refusing update from remote peer a419ade95f5446b8b7604b0e64e772c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:39.497347 2625 consensus_queue.cc:1035] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.497501 2667 consensus_queue.cc:1035] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:39.505455 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.502773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41773 (local address 127.0.98.3:57861)
0928 22:44:39.502915 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:39.502925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:39.502970 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:39.503106 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:39.503107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:39.503108 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:39.503150 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.503152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.504035 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.504036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.504576 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:39.504578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.505060 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.505061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:39.505122 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.505127 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.505312 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.505314 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:39.505358 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:44:39.505402 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 22:44:39.505460 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:39.502863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57861 (local address 127.0.98.2:41773)
0928 22:44:39.502906 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:39.502909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:39.502959 (+ 50us) server_negotiation.cc:407] Connection header received
0928 22:44:39.503009 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:39.503010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:39.503020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:39.503065 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:39.503206 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.503958 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.504665 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:39.504967 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:39.505031 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:39.505187 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:39.505222 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:39.505263 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:39.505365 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:44:39.505388 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:44:39.505837 2645 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:44:41.025512 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:41.025614 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:41.026378 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Sending DeleteTablet for 3 replicas of tablet 3b3ef1b32d354f18a48cb9f83828d896
I20260928 22:44:41.026624 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52226
I20260928 22:44:41.026695 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260928 22:44:41.026903 2702 tablet_replica.cc:330] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: stopping tablet replica
I20260928 22:44:41.026961 2702 raft_consensus.cc:2229] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:41.026942 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3ef1b32d354f18a48cb9f83828d896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260928 22:44:41.026983 2701 tablet_replica.cc:330] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: stopping tablet replica
I20260928 22:44:41.026997 2702 raft_consensus.cc:2258] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.027029 2701 raft_consensus.cc:2229] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:41.027061 2701 raft_consensus.cc:2258] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.027145 2703 tablet_replica.cc:330] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: stopping tablet replica
I20260928 22:44:41.027191 2703 raft_consensus.cc:2229] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:41.027241 2702 ts_tablet_manager.cc:1905] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.027276 2703 raft_consensus.cc:2258] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.027495 2703 ts_tablet_manager.cc:1905] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.028422 2701 ts_tablet_manager.cc:1905] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.028856 2702 ts_tablet_manager.cc:1918] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:44:41.028950 2702 log.cc:1196] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/3b3ef1b32d354f18a48cb9f83828d896
I20260928 22:44:41.029178 2702 ts_tablet_manager.cc:1939] T 3b3ef1b32d354f18a48cb9f83828d896 P 900073c787d640a0a75f5f5c001cd534: Deleting consensus metadata
I20260928 22:44:41.029531 2102 catalog_manager.cc:4841] TS 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373): tablet 3b3ef1b32d354f18a48cb9f83828d896 (table test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]) successfully deleted
I20260928 22:44:41.029959 2701 ts_tablet_manager.cc:1918] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:44:41.029965 2703 ts_tablet_manager.cc:1918] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:44:41.030010 2701 log.cc:1196] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/3b3ef1b32d354f18a48cb9f83828d896
I20260928 22:44:41.030012 2703 log.cc:1196] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/3b3ef1b32d354f18a48cb9f83828d896
I20260928 22:44:41.030185 2703 ts_tablet_manager.cc:1939] T 3b3ef1b32d354f18a48cb9f83828d896 P a419ade95f5446b8b7604b0e64e772c2: Deleting consensus metadata
I20260928 22:44:41.030193 2701 ts_tablet_manager.cc:1939] T 3b3ef1b32d354f18a48cb9f83828d896 P e487968695d24e7fb8be26034ad5e346: Deleting consensus metadata
I20260928 22:44:41.030728 2101 catalog_manager.cc:4841] TS e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207): tablet 3b3ef1b32d354f18a48cb9f83828d896 (table test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]) successfully deleted
I20260928 22:44:41.030802 2102 catalog_manager.cc:4841] TS a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773): tablet 3b3ef1b32d354f18a48cb9f83828d896 (table test-table [id=333bcc84c5b84c1e8ab5f8fc155716fc]) successfully deleted
Sep 28 22:44:41 dist-test-slave-pxr1 krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:44:41 dist-test-slave-pxr1 krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:41.054111 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.043404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.98.62:34191)
0928 22:44:41.043577 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.043579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.043586 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:41.043611 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.043612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.043621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.043676 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:41.044195 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.044756 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.045534 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.045766 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.045820 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.051928 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:41.051930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:41.051930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:41.051937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:41.053152 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:41.053449 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:41.053450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:41.053450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:41.053499 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:41.053632 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:41.053632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:41.053632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:41.053733 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:41.053803 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.053930 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.054009 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260928 22:44:41.055383 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:41.055528 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:41.056850 2531 tablet_service.cc:1467] Processing CreateTablet for tablet dc0afc2657334861a8e045eb3e83cb35 (DEFAULT_TABLE table=test-table [id=8d36e45611d74f6bbf0814314c2822c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.056854 2252 tablet_service.cc:1467] Processing CreateTablet for tablet dc0afc2657334861a8e045eb3e83cb35 (DEFAULT_TABLE table=test-table [id=8d36e45611d74f6bbf0814314c2822c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.056877 2392 tablet_service.cc:1467] Processing CreateTablet for tablet dc0afc2657334861a8e045eb3e83cb35 (DEFAULT_TABLE table=test-table [id=8d36e45611d74f6bbf0814314c2822c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.056985 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0afc2657334861a8e045eb3e83cb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.056982 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0afc2657334861a8e045eb3e83cb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.056983 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0afc2657334861a8e045eb3e83cb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.057943 2712 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Bootstrap starting.
I20260928 22:44:41.058279 2712 tablet_bootstrap.cc:654] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.058315 2713 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Bootstrap starting.
I20260928 22:44:41.058562 2714 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Bootstrap starting.
I20260928 22:44:41.058657 2713 tablet_bootstrap.cc:654] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.058959 2714 tablet_bootstrap.cc:654] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.059245 2713 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: No bootstrap required, opened a new log
I20260928 22:44:41.059298 2713 ts_tablet_manager.cc:1397] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:41.059464 2712 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: No bootstrap required, opened a new log
I20260928 22:44:41.059443 2713 raft_consensus.cc:348] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.059508 2713 raft_consensus.cc:374] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.059512 2712 ts_tablet_manager.cc:1397] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:41.059537 2713 raft_consensus.cc:729] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.059587 2713 consensus_queue.cc:260] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.059613 2712 raft_consensus.cc:348] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.059679 2712 raft_consensus.cc:374] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.059705 2712 raft_consensus.cc:729] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.059754 2712 consensus_queue.cc:260] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.059784 2713 ts_tablet_manager.cc:1428] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:41.059849 2712 ts_tablet_manager.cc:1428] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:41.060141 2714 tablet_bootstrap.cc:492] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: No bootstrap required, opened a new log
I20260928 22:44:41.060220 2714 ts_tablet_manager.cc:1397] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:41.060370 2714 raft_consensus.cc:348] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.060473 2714 raft_consensus.cc:374] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.060524 2714 raft_consensus.cc:729] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.060590 2714 consensus_queue.cc:260] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.060689 2714 ts_tablet_manager.cc:1428] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:41.080914 2718 raft_consensus.cc:482] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:41.081009 2718 raft_consensus.cc:504] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.081344 2718 leader_election.cc:288] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773), e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207)
I20260928 22:44:41.084183 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.081539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40401 (local address 127.0.98.2:41773)
0928 22:44:41.081692 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.081694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.081735 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:44:41.081817 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.081818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.081828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.081884 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:41.082011 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.082490 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083498 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.083870 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083931 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.083966 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.084070 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.084111 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260928 22:44:41.084204 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.081494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41773 (local address 127.0.98.1:40401)
0928 22:44:41.081679 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:41.081690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:41.081754 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:41.081898 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:41.081899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:41.081900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:41.081955 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.081957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.082563 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.082563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.083415 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.083417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.083941 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.084009 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.084013 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:41.084060 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:44:41.084102 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":46,"thread_start_us":121,"threads_started":1}
I20260928 22:44:41.084213 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.081591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39023 (local address 127.0.98.3:39207)
0928 22:44:41.081733 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.081735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.081744 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:41.081827 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.081828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.081837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.081900 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:41.082046 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.082508 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083500 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.083890 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083951 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.083964 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.084095 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.084132 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260928 22:44:41.084411 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.081500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.98.1:39023)
0928 22:44:41.081694 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:41.081705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:41.081757 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:41.081909 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:41.081909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:41.081911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:41.081978 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.081979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.082580 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.082580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.083416 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.083418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.083966 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.083967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.084033 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.084036 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:41.084080 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:44:41.084120 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":35,"thread_start_us":103,"threads_started":1}
I20260928 22:44:41.084555 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0afc2657334861a8e045eb3e83cb35" candidate_uuid: "900073c787d640a0a75f5f5c001cd534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2" is_pre_election: true
I20260928 22:44:41.084656 2412 raft_consensus.cc:2454] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900073c787d640a0a75f5f5c001cd534 in term 0.
I20260928 22:44:41.084693 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0afc2657334861a8e045eb3e83cb35" candidate_uuid: "900073c787d640a0a75f5f5c001cd534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e487968695d24e7fb8be26034ad5e346" is_pre_election: true
I20260928 22:44:41.084802 2552 raft_consensus.cc:2454] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900073c787d640a0a75f5f5c001cd534 in term 0.
I20260928 22:44:41.084846 2206 leader_election.cc:302] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, a419ade95f5446b8b7604b0e64e772c2; no voters:
I20260928 22:44:41.084990 2718 raft_consensus.cc:2781] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:41.085036 2718 raft_consensus.cc:482] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:41.085057 2718 raft_consensus.cc:3037] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.085700 2718 raft_consensus.cc:504] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.085809 2718 leader_election.cc:288] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [CANDIDATE]: Term 1 election: Requested vote from peers a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773), e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207)
I20260928 22:44:41.085966 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0afc2657334861a8e045eb3e83cb35" candidate_uuid: "900073c787d640a0a75f5f5c001cd534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e487968695d24e7fb8be26034ad5e346"
I20260928 22:44:41.085965 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0afc2657334861a8e045eb3e83cb35" candidate_uuid: "900073c787d640a0a75f5f5c001cd534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2"
I20260928 22:44:41.086030 2552 raft_consensus.cc:3037] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.086031 2412 raft_consensus.cc:3037] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.086577 2552 raft_consensus.cc:2454] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900073c787d640a0a75f5f5c001cd534 in term 1.
I20260928 22:44:41.086620 2412 raft_consensus.cc:2454] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900073c787d640a0a75f5f5c001cd534 in term 1.
I20260928 22:44:41.086755 2205 leader_election.cc:302] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:41.086869 2718 raft_consensus.cc:2781] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:41.087002 2718 raft_consensus.cc:686] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 1 LEADER]: Becoming Leader. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Running, Role: LEADER
I20260928 22:44:41.087060 2718 consensus_queue.cc:237] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.087659 2115 catalog_manager.cc:5495] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 900073c787d640a0a75f5f5c001cd534 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "900073c787d640a0a75f5f5c001cd534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:41.098179 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.095976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.98.1:43373)
0928 22:44:41.096121 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.096122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.096130 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:41.096159 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.096160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.096178 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.096251 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:41.096336 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.096870 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.097545 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.097749 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.097796 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.097909 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.098008 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.098056 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.098071 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.098085 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260928 22:44:41.102474 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.100340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41063 (local address 127.0.98.3:39207)
0928 22:44:41.100384 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.100386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.100409 (+ 23us) server_negotiation.cc:407] Connection header received
0928 22:44:41.100468 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.100469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.100480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.100525 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:41.100646 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.101141 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.101852 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.102051 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.102103 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.102258 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.102291 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.102330 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.102378 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.102400 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:41.102495 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.100217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39207 (local address 127.0.98.1:41063)
0928 22:44:41.100356 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:41.100367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:41.100423 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:41.100536 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:41.100537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:41.100538 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:41.100592 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.100594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.101211 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.101212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.101780 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.101781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.102109 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.102110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.102171 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.102192 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.102336 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.102338 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:41.102376 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:44:41.102417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 22:44:41.104451 2412 raft_consensus.cc:1261] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Refusing update from remote peer 900073c787d640a0a75f5f5c001cd534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:41.104496 2552 raft_consensus.cc:1261] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Refusing update from remote peer 900073c787d640a0a75f5f5c001cd534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:41.104679 2718 consensus_queue.cc:1035] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:41.104846 2723 consensus_queue.cc:1035] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:41.111363 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.108984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49313 (local address 127.0.98.1:43373)
0928 22:44:41.109022 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.109025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.109135 (+ 110us) server_negotiation.cc:407] Connection header received
0928 22:44:41.109220 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.109221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.109229 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.109267 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:41.109345 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.109787 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.110580 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.110853 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.110911 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.111123 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.111169 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.111211 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.111267 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.111290 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:41.111380 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.108951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43373 (local address 127.0.98.3:49313)
0928 22:44:41.109069 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:41.109079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:41.109132 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:41.109271 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:41.109272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:41.109273 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:44:41.109311 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.109312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.109871 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.109872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.110499 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:41.110501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.110922 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.110923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:41.110978 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.110983 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.111225 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.111226 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:44:41.111266 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:44:41.111310 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260928 22:44:41.118126 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:41.118227 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:41.118789 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Sending DeleteTablet for 3 replicas of tablet dc0afc2657334861a8e045eb3e83cb35
I20260928 22:44:41.119007 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0afc2657334861a8e045eb3e83cb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52226
I20260928 22:44:41.119068 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0afc2657334861a8e045eb3e83cb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260928 22:44:41.119100 2701 tablet_replica.cc:330] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: stopping tablet replica
I20260928 22:44:41.119148 2701 raft_consensus.cc:2229] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:41.119153 2703 tablet_replica.cc:330] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: stopping tablet replica
I20260928 22:44:41.119181 2701 raft_consensus.cc:2258] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.119204 2703 raft_consensus.cc:2229] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:41.119236 2703 raft_consensus.cc:2258] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.119231 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:41.119374 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:41.119376 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0afc2657334861a8e045eb3e83cb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260928 22:44:41.119495 2701 ts_tablet_manager.cc:1905] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.119517 2702 tablet_replica.cc:330] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: stopping tablet replica
I20260928 22:44:41.119561 2702 raft_consensus.cc:2229] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:41.119625 2702 raft_consensus.cc:2258] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:41.119879 2702 ts_tablet_manager.cc:1905] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.121126 2701 ts_tablet_manager.cc:1918] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:41.121206 2701 log.cc:1196] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/dc0afc2657334861a8e045eb3e83cb35
I20260928 22:44:41.121281 2702 ts_tablet_manager.cc:1918] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:41.121318 2702 log.cc:1196] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/dc0afc2657334861a8e045eb3e83cb35
I20260928 22:44:41.121450 2702 ts_tablet_manager.cc:1939] T dc0afc2657334861a8e045eb3e83cb35 P 900073c787d640a0a75f5f5c001cd534: Deleting consensus metadata
I20260928 22:44:41.121435 2701 ts_tablet_manager.cc:1939] T dc0afc2657334861a8e045eb3e83cb35 P e487968695d24e7fb8be26034ad5e346: Deleting consensus metadata
I20260928 22:44:41.121707 2102 catalog_manager.cc:4841] TS 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373): tablet dc0afc2657334861a8e045eb3e83cb35 (table test-table [id=8d36e45611d74f6bbf0814314c2822c4]) successfully deleted
I20260928 22:44:41.121729 2101 catalog_manager.cc:4841] TS e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207): tablet dc0afc2657334861a8e045eb3e83cb35 (table test-table [id=8d36e45611d74f6bbf0814314c2822c4]) successfully deleted
I20260928 22:44:41.122265 2703 ts_tablet_manager.cc:1905] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:41.122928 2531 tablet_service.cc:1467] Processing CreateTablet for tablet e259f995027d4568b1a069c993296654 (DEFAULT_TABLE table=test-table [id=91318b323a094a44aca73aacea3f1d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.122917 2392 tablet_service.cc:1467] Processing CreateTablet for tablet e259f995027d4568b1a069c993296654 (DEFAULT_TABLE table=test-table [id=91318b323a094a44aca73aacea3f1d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.123013 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259f995027d4568b1a069c993296654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.123078 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259f995027d4568b1a069c993296654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.123998 2703 ts_tablet_manager.cc:1918] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:44:41.124053 2703 log.cc:1196] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/dc0afc2657334861a8e045eb3e83cb35
I20260928 22:44:41.124244 2703 ts_tablet_manager.cc:1939] T dc0afc2657334861a8e045eb3e83cb35 P a419ade95f5446b8b7604b0e64e772c2: Deleting consensus metadata
I20260928 22:44:41.124238 2714 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Bootstrap starting.
I20260928 22:44:41.124507 2714 tablet_bootstrap.cc:654] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.124752 2712 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Bootstrap starting.
I20260928 22:44:41.125042 2712 tablet_bootstrap.cc:654] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.125043 2714 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: No bootstrap required, opened a new log
I20260928 22:44:41.125085 2714 ts_tablet_manager.cc:1397] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:41.125218 2714 raft_consensus.cc:348] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.125283 2714 raft_consensus.cc:374] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.125303 2714 raft_consensus.cc:729] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.125372 2714 consensus_queue.cc:260] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.125383 2252 tablet_service.cc:1467] Processing CreateTablet for tablet e259f995027d4568b1a069c993296654 (DEFAULT_TABLE table=test-table [id=91318b323a094a44aca73aacea3f1d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:41.125474 2714 ts_tablet_manager.cc:1428] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:41.125484 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259f995027d4568b1a069c993296654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:41.125506 2712 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: No bootstrap required, opened a new log
I20260928 22:44:41.125541 2712 ts_tablet_manager.cc:1397] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:41.125645 2712 raft_consensus.cc:348] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.125725 2712 raft_consensus.cc:374] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.125741 2712 raft_consensus.cc:729] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a419ade95f5446b8b7604b0e64e772c2, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.125773 2712 consensus_queue.cc:260] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.125844 2712 ts_tablet_manager.cc:1428] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:41.126648 2713 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534: Bootstrap starting.
I20260928 22:44:41.126770 2102 catalog_manager.cc:4841] TS a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773): tablet dc0afc2657334861a8e045eb3e83cb35 (table test-table [id=8d36e45611d74f6bbf0814314c2822c4]) successfully deleted
I20260928 22:44:41.127046 2713 tablet_bootstrap.cc:654] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:41.127595 2713 tablet_bootstrap.cc:492] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534: No bootstrap required, opened a new log
I20260928 22:44:41.127635 2713 ts_tablet_manager.cc:1397] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:41.127779 2713 raft_consensus.cc:348] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.127830 2713 raft_consensus.cc:374] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:41.127842 2713 raft_consensus.cc:729] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900073c787d640a0a75f5f5c001cd534, State: Initialized, Role: FOLLOWER
I20260928 22:44:41.127884 2713 consensus_queue.cc:260] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.127996 2713 ts_tablet_manager.cc:1428] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:41.175802 2682 raft_consensus.cc:482] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:41.175901 2682 raft_consensus.cc:504] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.176086 2682 leader_election.cc:288] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:41.176316 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259f995027d4568b1a069c993296654" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534" is_pre_election: true
I20260928 22:44:41.176443 2272 raft_consensus.cc:2454] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:41.176499 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259f995027d4568b1a069c993296654" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2" is_pre_election: true
I20260928 22:44:41.176586 2412 raft_consensus.cc:2454] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 0.
I20260928 22:44:41.176636 2486 leader_election.cc:302] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:41.176751 2682 raft_consensus.cc:2781] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:41.176811 2682 raft_consensus.cc:482] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:41.176839 2682 raft_consensus.cc:3037] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.177731 2682 raft_consensus.cc:504] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.177870 2682 leader_election.cc:288] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Requested vote from peers 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373), a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773)
I20260928 22:44:41.178014 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259f995027d4568b1a069c993296654" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900073c787d640a0a75f5f5c001cd534"
I20260928 22:44:41.178012 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259f995027d4568b1a069c993296654" candidate_uuid: "e487968695d24e7fb8be26034ad5e346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a419ade95f5446b8b7604b0e64e772c2"
I20260928 22:44:41.178085 2412 raft_consensus.cc:3037] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.178085 2272 raft_consensus.cc:3037] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:41.178781 2272 raft_consensus.cc:2454] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:41.178781 2412 raft_consensus.cc:2454] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e487968695d24e7fb8be26034ad5e346 in term 1.
I20260928 22:44:41.178949 2486 leader_election.cc:302] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900073c787d640a0a75f5f5c001cd534, e487968695d24e7fb8be26034ad5e346; no voters:
I20260928 22:44:41.179044 2682 raft_consensus.cc:2781] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:41.179095 2682 raft_consensus.cc:686] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 1 LEADER]: Becoming Leader. State: Replica: e487968695d24e7fb8be26034ad5e346, State: Running, Role: LEADER
I20260928 22:44:41.179143 2682 consensus_queue.cc:237] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } }
I20260928 22:44:41.179715 2115 catalog_manager.cc:5495] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 reported cstate change: term changed from 0 to 1, leader changed from <none> to e487968695d24e7fb8be26034ad5e346 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e487968695d24e7fb8be26034ad5e346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e487968695d24e7fb8be26034ad5e346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39207 } health_report { overall_health: HEALTHY } } }
I20260928 22:44:41.188431 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:41.186147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.0.98.3:39207)
0928 22:44:41.186185 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:41.186187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:41.186196 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:41.186225 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:41.186226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:41.186236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:41.186303 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:41.186390 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.187100 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.187778 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:41.188055 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:41.188135 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:41.188209 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:41.188241 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:41.188283 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:41.188322 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 22:44:41.188345 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:41.190692 2412 raft_consensus.cc:1261] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:41.190764 2271 raft_consensus.cc:1261] T e259f995027d4568b1a069c993296654 P 900073c787d640a0a75f5f5c001cd534 [term 1 FOLLOWER]: Refusing update from remote peer e487968695d24e7fb8be26034ad5e346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:41.190887 2681 consensus_queue.cc:1035] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a419ade95f5446b8b7604b0e64e772c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:41.191151 2681 consensus_queue.cc:1035] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900073c787d640a0a75f5f5c001cd534" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:44:41.195746 2679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:44:42.714485 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:42.714582 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:42.715380 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27fb123c86c42939733f3193d17280a: Sending DeleteTablet for 3 replicas of tablet e259f995027d4568b1a069c993296654
I20260928 22:44:42.715600 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet e259f995027d4568b1a069c993296654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52226
I20260928 22:44:42.715629 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet e259f995027d4568b1a069c993296654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260928 22:44:42.715677 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet e259f995027d4568b1a069c993296654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260928 22:44:42.715714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2187
I20260928 22:44:42.716425 2765 tablet_replica.cc:330] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: stopping tablet replica
I20260928 22:44:42.716557 2765 raft_consensus.cc:2229] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:42.716702 2765 raft_consensus.cc:2258] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:42.716935 2763 tablet_replica.cc:330] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: stopping tablet replica
I20260928 22:44:42.717031 2765 ts_tablet_manager.cc:1905] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:42.717033 2763 raft_consensus.cc:2229] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:42.717127 2763 raft_consensus.cc:2258] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:42.717389 2763 ts_tablet_manager.cc:1905] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:42.718720 2765 ts_tablet_manager.cc:1918] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:44:42.718786 2765 log.cc:1196] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/e259f995027d4568b1a069c993296654
I20260928 22:44:42.718883 2763 ts_tablet_manager.cc:1918] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:44:42.718935 2763 log.cc:1196] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/e259f995027d4568b1a069c993296654
I20260928 22:44:42.718979 2765 ts_tablet_manager.cc:1939] T e259f995027d4568b1a069c993296654 P a419ade95f5446b8b7604b0e64e772c2: Deleting consensus metadata
I20260928 22:44:42.719135 2763 ts_tablet_manager.cc:1939] T e259f995027d4568b1a069c993296654 P e487968695d24e7fb8be26034ad5e346: Deleting consensus metadata
I20260928 22:44:42.719316 2102 catalog_manager.cc:4841] TS a419ade95f5446b8b7604b0e64e772c2 (127.0.98.2:41773): tablet e259f995027d4568b1a069c993296654 (table test-table [id=91318b323a094a44aca73aacea3f1d6a]) successfully deleted
I20260928 22:44:42.719502 2101 catalog_manager.cc:4841] TS e487968695d24e7fb8be26034ad5e346 (127.0.98.3:39207): tablet e259f995027d4568b1a069c993296654 (table test-table [id=91318b323a094a44aca73aacea3f1d6a]) successfully deleted
W20260928 22:44:42.723054 2102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43373 (error 108)
I20260928 22:44:42.723284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2327
I20260928 22:44:42.724047 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:42.723430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43373 (local address 127.0.0.1:58514)
0928 22:44:42.723957 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:42.724006 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":56,"threads_started":1}
W20260928 22:44:42.724133 2102 catalog_manager.cc:4584] TS 900073c787d640a0a75f5f5c001cd534 (127.0.98.1:43373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e259f995027d4568b1a069c993296654: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43373: connect: Connection refused (error 111)
I20260928 22:44:42.728297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2467
I20260928 22:44:42.733291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2083
2026-09-28T22:44:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5542 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:42 dist-test-slave-pxr1 krb5kdc[2770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:42 dist-test-slave-pxr1 krb5kdc[2770](info): set up 2 sockets
Sep 28 22:44:42 dist-test-slave-pxr1 krb5kdc[2770](info): commencing operation
krb5kdc: starting...
W20260928 22:44:43.688346 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:43 dist-test-slave-pxr1 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:43.808030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:43.887399 2783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:43.887610 2783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:43.887639 2783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:43.888844 2783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:43.888885 2783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:43.888903 2783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:43.888919 2783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:43.888933 2783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:43.890488 2783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:43.890740 2783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:43.890955 2783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:43.895614 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:43.895608 2789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:43.895608 2791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:43.895846 2783 server_base.cc:1034] running on GCE node
I20260928 22:44:43.896133 2783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:43.896404 2783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:43.897567 2783 hybrid_clock.cc:648] HybridClock initialized: now 1790635483897552 us; error 26 us; skew 500 ppm
Sep 28 22:44:43 dist-test-slave-pxr1 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:43.900194 2783 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:43.900632 2783 webserver.cc:460] Webserver started at http://127.0.98.62:35017/ using document root <none> and password file <none>
I20260928 22:44:43.900880 2783 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:43.900929 2783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:43.901041 2783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:43.902098 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "42b2e3b3242d46668f554ea44c74c552"
format_stamp: "Formatted at 2026-09-28 22:44:43 on dist-test-slave-pxr1"
server_key: "feff8760aef7191d630d81cf0d7e4b76"
server_key_iv: "945a01f5f9bc1ba9f32fb524ede1c694"
server_key_version: "encryptionkey@0"
I20260928 22:44:43.902421 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "42b2e3b3242d46668f554ea44c74c552"
format_stamp: "Formatted at 2026-09-28 22:44:43 on dist-test-slave-pxr1"
server_key: "feff8760aef7191d630d81cf0d7e4b76"
server_key_iv: "945a01f5f9bc1ba9f32fb524ede1c694"
server_key_version: "encryptionkey@0"
I20260928 22:44:43.903761 2783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:43.904556 2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:43.904876 2783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:43.904973 2783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "42b2e3b3242d46668f554ea44c74c552"
format_stamp: "Formatted at 2026-09-28 22:44:43 on dist-test-slave-pxr1"
server_key: "feff8760aef7191d630d81cf0d7e4b76"
server_key_iv: "945a01f5f9bc1ba9f32fb524ede1c694"
server_key_version: "encryptionkey@0"
I20260928 22:44:43.905048 2783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:43.932313 2783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:43.934763 2783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:43.934911 2783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:43.938717 2783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36357
I20260928 22:44:43.938750 2850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36357 every 8 connection(s)
I20260928 22:44:43.939245 2783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:43.939895 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:43.942255 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552: Bootstrap starting.
I20260928 22:44:43.942828 2851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:43.942935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2783
I20260928 22:44:43.943030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:43.943128 2851 log.cc:824] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:43.943200 392 external_mini_cluster.cc:1428] Setting key d4d5ad4a84dd33374927abe52754615c
I20260928 22:44:43.943840 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552: No bootstrap required, opened a new log
I20260928 22:44:43.945991 2851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } }
I20260928 22:44:43.946146 2851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:43.946179 2851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b2e3b3242d46668f554ea44c74c552, State: Initialized, Role: FOLLOWER
I20260928 22:44:43.946305 2851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } }
I20260928 22:44:43.946382 2851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:43.946421 2851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:43.946487 2851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:43.947029 2851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } }
I20260928 22:44:43.947136 2851 leader_election.cc:302] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42b2e3b3242d46668f554ea44c74c552; no voters:
I20260928 22:44:43.947297 2851 leader_election.cc:288] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:43.947333 2856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:43.947433 2856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [term 1 LEADER]: Becoming Leader. State: Replica: 42b2e3b3242d46668f554ea44c74c552, State: Running, Role: LEADER
I20260928 22:44:43.947571 2851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:43.947592 2856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } }
I20260928 22:44:43.947973 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42b2e3b3242d46668f554ea44c74c552. Latest consensus state: current_term: 1 leader_uuid: "42b2e3b3242d46668f554ea44c74c552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } } }
I20260928 22:44:43.948000 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42b2e3b3242d46668f554ea44c74c552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b2e3b3242d46668f554ea44c74c552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36357 } } }
I20260928 22:44:43.948141 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:43.948151 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:43.948374 2865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:43.948949 2865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:44:43 dist-test-slave-pxr1 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:43.954133 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:43.943676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.0.98.62:36357)
0928 22:44:43.943854 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:43.943862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:43.943887 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:44:43.944339 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:43.944346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:43.944397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:43.944538 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:43.945192 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:43.946073 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:43.946925 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:43.947407 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:43.947525 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:43.950808 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:43.950812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:43.950812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:43.950823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:43.952371 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:43.952802 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:43.952805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:43.952805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:43.952918 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:43.953125 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:43.953126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:43.953126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:43.953398 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:43.953538 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:43.953658 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 22:44:43.953766 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 22:44:43.954190 2865 catalog_manager.cc:1347] Generated new cluster ID: 6a6ecef4c41948f582e73062edad2c89
I20260928 22:44:43.954237 2865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:43.961206 2865 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:43.961782 2865 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:43.967681 2865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42b2e3b3242d46668f554ea44c74c552: Generated new TSK 0
I20260928 22:44:43.967904 2865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:43.999754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:44.110466 2877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:44.110700 2877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:44.110725 2877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:44.111946 2877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:44.111994 2877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:44.112021 2877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:44.113619 2877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:44.113827 2877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:44.114020 2877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:44.118870 2885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:44.118949 2877 server_base.cc:1034] running on GCE node
W20260928 22:44:44.118875 2882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:44.118888 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:44.119289 2877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:44.119516 2877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:44.120647 2877 hybrid_clock.cc:648] HybridClock initialized: now 1790635484120639 us; error 24 us; skew 500 ppm
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:44.123013 2877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:44.123431 2877 webserver.cc:460] Webserver started at http://127.0.98.1:33405/ using document root <none> and password file <none>
I20260928 22:44:44.123634 2877 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:44.123680 2877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:44.123798 2877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:44.124869 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "d60accc6bb824d529f33cb40f9448871"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "fbec88bc903bfef0b839301215caea32"
server_key_iv: "28a0413eb3f842875fad396318f63147"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.125212 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d60accc6bb824d529f33cb40f9448871"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "fbec88bc903bfef0b839301215caea32"
server_key_iv: "28a0413eb3f842875fad396318f63147"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.126411 2877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:44.127081 2892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.127234 2877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:44.127311 2877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "d60accc6bb824d529f33cb40f9448871"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "fbec88bc903bfef0b839301215caea32"
server_key_iv: "28a0413eb3f842875fad396318f63147"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.127369 2877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:44.163138 2877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:44.165331 2877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:44.165472 2877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:44.165695 2877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:44.165987 2877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:44.166028 2877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.166056 2877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:44.166079 2877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.171289 2877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38255
I20260928 22:44:44.171332 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38255 every 8 connection(s)
I20260928 22:44:44.171649 2877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:44.175869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 2877
I20260928 22:44:44.175962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:44.176143 392 external_mini_cluster.cc:1428] Setting key d1c6a296ba11d4da92131a383fe0c018
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:44.180923 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.172347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45345 (local address 127.0.98.62:36357)
0928 22:44:44.172417 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:44.172419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:44.172613 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:44:44.173568 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:44.173569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:44.173579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:44.173641 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:44.174227 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.174719 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.175666 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.175856 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.175899 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.178384 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:44.178385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:44.178386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:44.178393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:44.179651 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.180012 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.180013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.180013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.180064 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.180287 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.180287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.180288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.180392 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:44.180454 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:44.180740 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 22:44:44.180831 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:44:44.181073 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.172312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36357 (local address 127.0.98.1:45345)
0928 22:44:44.172489 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:44.172531 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:44.173479 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:44.173708 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:44.173712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:44.173959 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:44.174162 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.174167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.174780 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.174780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.175604 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.175606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.175930 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.175932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.176079 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.176749 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:44.176759 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:44.178289 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:44.179728 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.179735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.179737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.179949 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.180124 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.180125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.180125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.180232 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.180450 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:44.180456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:44.180600 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 22:44:44.180686 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:44:44.180861 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260928 22:44:44.181542 3006 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36357
I20260928 22:44:44.181630 3006 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:44.181798 3006 heartbeater.cc:511] Master 127.0.98.62:36357 requested a full tablet report, sending...
I20260928 22:44:44.182294 2815 ts_manager.cc:194] Registered new tserver with Master: d60accc6bb824d529f33cb40f9448871 (127.0.98.1:38255)
I20260928 22:44:44.182883 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45345
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:44.199832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:44.279031 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:44.279248 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:44.279273 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:44.280465 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:44.280519 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:44.280547 3011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:44.282146 3011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:44.282382 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:44.282601 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:44.287528 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:44.287514 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:44.287514 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:44.287843 3011 server_base.cc:1034] running on GCE node
I20260928 22:44:44.288059 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:44.288272 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:44.289728 3011 hybrid_clock.cc:648] HybridClock initialized: now 1790635484289718 us; error 39 us; skew 500 ppm
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:44.292399 3011 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:44.292917 3011 webserver.cc:460] Webserver started at http://127.0.98.2:35669/ using document root <none> and password file <none>
I20260928 22:44:44.293140 3011 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:44.293190 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:44.293329 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:44.294377 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "20dc487b85bd4870ba23c0f11b7972fc"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f5120b2d37d946465b57ebfec768569c"
server_key_iv: "46248733ffbe0ff1ea73fce5732f5070"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.294709 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20dc487b85bd4870ba23c0f11b7972fc"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f5120b2d37d946465b57ebfec768569c"
server_key_iv: "46248733ffbe0ff1ea73fce5732f5070"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.295982 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:44.296695 3026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.296875 3011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:44.296958 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "20dc487b85bd4870ba23c0f11b7972fc"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f5120b2d37d946465b57ebfec768569c"
server_key_iv: "46248733ffbe0ff1ea73fce5732f5070"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.297031 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:44.312443 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:44.314873 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:44.315035 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:44.315268 3011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:44.315690 3011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:44.315747 3011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.315783 3011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:44.315799 3011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.321744 3011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41597
I20260928 22:44:44.321827 3139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41597 every 8 connection(s)
I20260928 22:44:44.322124 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:44.324633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3011
I20260928 22:44:44.324754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:44.324951 392 external_mini_cluster.cc:1428] Setting key df3821071df36c6c717dc1d4ed427cb6
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:44.332350 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.322832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43105 (local address 127.0.98.62:36357)
0928 22:44:44.322874 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:44.322876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:44.323088 (+ 212us) server_negotiation.cc:407] Connection header received
0928 22:44:44.324015 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:44.324017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:44.324027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:44.324094 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:44.324742 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.325397 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.326594 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.326870 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.326928 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.329665 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:44.329667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:44.329668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:44.329676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:44.331033 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.331431 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.331432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.331433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.331493 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.331719 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.331719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.331720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.331839 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:44.331895 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:44.332167 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 22:44:44.332252 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 22:44:44.332557 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.322817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36357 (local address 127.0.98.2:43105)
0928 22:44:44.322954 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:44.322989 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:44.323902 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:44.324164 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:44.324167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:44.324423 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:44.324619 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.324623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.325500 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.325502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.326482 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.326485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.326955 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.326956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.327114 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.327834 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:44.327845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:44.329556 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:44.331117 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.331126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.331129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.331360 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.331548 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.331549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.331549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.331663 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.331892 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:44.331899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:44.332047 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:44:44.332140 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:44:44.332307 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 22:44:44.333052 3140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36357
I20260928 22:44:44.333153 3140 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:44.333316 3140 heartbeater.cc:511] Master 127.0.98.62:36357 requested a full tablet report, sending...
I20260928 22:44:44.333705 2815 ts_manager.cc:194] Registered new tserver with Master: 20dc487b85bd4870ba23c0f11b7972fc (127.0.98.2:41597)
I20260928 22:44:44.334072 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:44.352013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:44.430833 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:44.431064 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:44.431092 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:44.432268 3145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:44.432314 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:44.432340 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:44.433878 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:44.434103 3145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:44.434329 3145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:44.439118 3153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:44.439124 3150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:44.439116 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:44.439376 3145 server_base.cc:1034] running on GCE node
I20260928 22:44:44.439632 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:44.439894 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:44.441054 3145 hybrid_clock.cc:648] HybridClock initialized: now 1790635484441038 us; error 34 us; skew 500 ppm
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:44.443486 3145 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:44.443922 3145 webserver.cc:460] Webserver started at http://127.0.98.3:32867/ using document root <none> and password file <none>
I20260928 22:44:44.444106 3145 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:44.444145 3145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:44.444238 3145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:44.445272 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "ded3bf991cda4a33b2a8805887298f03"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f2553f0c2f61fd95a719b59c60460547"
server_key_iv: "0d2980f68d8a0fce39d6f850f2ad603d"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.445587 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ded3bf991cda4a33b2a8805887298f03"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f2553f0c2f61fd95a719b59c60460547"
server_key_iv: "0d2980f68d8a0fce39d6f850f2ad603d"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.446913 3145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:44.447814 3160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.447968 3145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:44.448055 3145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "ded3bf991cda4a33b2a8805887298f03"
format_stamp: "Formatted at 2026-09-28 22:44:44 on dist-test-slave-pxr1"
server_key: "f2553f0c2f61fd95a719b59c60460547"
server_key_iv: "0d2980f68d8a0fce39d6f850f2ad603d"
server_key_version: "encryptionkey@0"
I20260928 22:44:44.448122 3145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:44.464591 3145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:44.467031 3145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:44.467181 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:44.467404 3145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:44.467773 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:44.467823 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.467861 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:44.467885 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:44.473235 3145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37223
I20260928 22:44:44.473302 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37223 every 8 connection(s)
I20260928 22:44:44.473600 3145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:44.476861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3145
I20260928 22:44:44.476935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:44.477080 392 external_mini_cluster.cc:1428] Setting key d87f1526054bd7bf8d339fb64a6c2f6d
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:44.482467 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.474255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52027 (local address 127.0.98.62:36357)
0928 22:44:44.474316 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:44.474318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:44.474513 (+ 195us) server_negotiation.cc:407] Connection header received
0928 22:44:44.475467 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:44.475468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:44.475481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:44.475539 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:44.476177 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.476690 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.477975 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.478176 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.478230 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.480199 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:44.480200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:44.480200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:44.480206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:44.481379 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.481761 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.481762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.481762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.481805 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:44.481991 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:44.481991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:44.481991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:44.482093 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:44.482136 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:44.482346 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 22:44:44.482403 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:44:44.482684 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:44.474233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36357 (local address 127.0.98.3:52027)
0928 22:44:44.474394 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:44.474431 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:44.475356 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:44.475617 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:44.475621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:44.475882 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:44.476082 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.476088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.476810 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.476812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.477907 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:44.477910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:44.478236 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:44.478237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:44.478332 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:44.478850 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:44.478858 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:44.480121 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:44.481458 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.481464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.481466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.481700 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.481850 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:44.481851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:44.481851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:44.481949 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:44.482133 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:44.482136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:44.482250 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 22:44:44.482328 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:44:44.482462 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 22:44:44.483099 3274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36357
I20260928 22:44:44.483194 3274 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:44.483378 3274 heartbeater.cc:511] Master 127.0.98.62:36357 requested a full tablet report, sending...
I20260928 22:44:44.483736 2815 ts_manager.cc:194] Registered new tserver with Master: ded3bf991cda4a33b2a8805887298f03 (127.0.98.3:37223)
I20260928 22:44:44.484107 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52027
I20260928 22:44:44.488870 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:44:44.499176 2855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:44.498267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.0.98.62:36357)
0928 22:44:44.498330 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:44.498332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:44.498408 (+ 76us) server_negotiation.cc:407] Connection header received
0928 22:44:44.498464 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:44.498465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:44.498474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:44.498529 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:44.499103 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54784: BlockingRecv error: recv got EOF from 127.0.0.1:54784 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:44:44.500006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2877
I20260928 22:44:44.503896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3011
I20260928 22:44:44.507647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3145
I20260928 22:44:44.511255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 2783
2026-09-28T22:44:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1776 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[3286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[3286](info): set up 2 sockets
Sep 28 22:44:44 dist-test-slave-pxr1 krb5kdc[3286](info): commencing operation
krb5kdc: starting...
W20260928 22:44:45.456315 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:45.568620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:45.648170 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:45.648372 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:45.648394 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:45.649534 3299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:45.649572 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:45.649585 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:45.649596 3299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:45.649606 3299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:45.651070 3299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:45.651261 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:45.651448 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:45.656102 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:45.656157 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:45.656232 3299 server_base.cc:1034] running on GCE node
W20260928 22:44:45.656169 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:45.656591 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:45.656909 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:45.658046 3299 hybrid_clock.cc:648] HybridClock initialized: now 1790635485658034 us; error 25 us; skew 500 ppm
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:45.660633 3299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:45.661150 3299 webserver.cc:460] Webserver started at http://127.0.98.62:35735/ using document root <none> and password file <none>
I20260928 22:44:45.661367 3299 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:45.661415 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:45.661540 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:45.662688 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "33337e6c8d38448c9ece3bd38252931d"
format_stamp: "Formatted at 2026-09-28 22:44:45 on dist-test-slave-pxr1"
server_key: "0c6ae7da12f7514021360acbce796c22"
server_key_iv: "1d55774c94902c4ffa6815b65befcfe7"
server_key_version: "encryptionkey@0"
I20260928 22:44:45.663020 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "33337e6c8d38448c9ece3bd38252931d"
format_stamp: "Formatted at 2026-09-28 22:44:45 on dist-test-slave-pxr1"
server_key: "0c6ae7da12f7514021360acbce796c22"
server_key_iv: "1d55774c94902c4ffa6815b65befcfe7"
server_key_version: "encryptionkey@0"
I20260928 22:44:45.664278 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:44:45.664999 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:45.665138 3299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:45.665222 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "33337e6c8d38448c9ece3bd38252931d"
format_stamp: "Formatted at 2026-09-28 22:44:45 on dist-test-slave-pxr1"
server_key: "0c6ae7da12f7514021360acbce796c22"
server_key_iv: "1d55774c94902c4ffa6815b65befcfe7"
server_key_version: "encryptionkey@0"
I20260928 22:44:45.665287 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:45.686998 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:45.689168 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:45.689302 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:45.692848 3299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41745
I20260928 22:44:45.692904 3366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41745 every 8 connection(s)
I20260928 22:44:45.693229 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:45.694026 3367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:45.696272 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d: Bootstrap starting.
I20260928 22:44:45.696853 3367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:45.697126 3367 log.cc:824] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:45.697798 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d: No bootstrap required, opened a new log
I20260928 22:44:45.699369 3367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260928 22:44:45.699510 3367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:45.699541 3367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33337e6c8d38448c9ece3bd38252931d, State: Initialized, Role: FOLLOWER
I20260928 22:44:45.699683 3367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260928 22:44:45.699743 3367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:45.699771 3367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:45.699819 3367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:45.700599 3367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260928 22:44:45.700696 3367 leader_election.cc:302] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33337e6c8d38448c9ece3bd38252931d; no voters:
I20260928 22:44:45.700894 3367 leader_election.cc:288] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:45.700941 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:45.701061 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [term 1 LEADER]: Becoming Leader. State: Replica: 33337e6c8d38448c9ece3bd38252931d, State: Running, Role: LEADER
I20260928 22:44:45.701172 3367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:45.701205 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } }
I20260928 22:44:45.701560 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33337e6c8d38448c9ece3bd38252931d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } } }
I20260928 22:44:45.701606 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33337e6c8d38448c9ece3bd38252931d. Latest consensus state: current_term: 1 leader_uuid: "33337e6c8d38448c9ece3bd38252931d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33337e6c8d38448c9ece3bd38252931d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41745 } } }
I20260928 22:44:45.701733 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:45.702078 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:45.702178 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:45.702628 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:45.703203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3299
I20260928 22:44:45.703289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:45.703471 392 external_mini_cluster.cc:1428] Setting key 2640cdf038dd7b6a0b1c20e1e4534608
I20260928 22:44:45.704340 3379 catalog_manager.cc:1347] Generated new cluster ID: ccc4db0092e04c4486ff9ecd513af157
I20260928 22:44:45.704392 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:45.712334 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:45.703908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.0.98.62:41745)
0928 22:44:45.704070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:45.704080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:45.704115 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:44:45.704234 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:45.704241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:45.704302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:45.704469 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:45.705115 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:45.706045 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:45.706805 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:45.707241 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:45.707335 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:45.709498 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:45.709502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:45.709502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:45.709514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:45.710963 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:45.711278 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:45.711281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:45.711281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:45.711363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:45.711536 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:45.711537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:45.711537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:45.711764 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:45.711911 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:45.712004 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:44:45.712144 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 22:44:45.713181 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:44:45.720901 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:45.721592 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:45.725401 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:45.722272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.62:41745)
0928 22:44:45.722310 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:45.722312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:45.722427 (+ 115us) server_negotiation.cc:407] Connection header received
0928 22:44:45.722486 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:45.722486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:45.722495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:45.722556 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:45.722881 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:45.723355 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:45.724060 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:45.724276 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:45.724330 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:45.724997 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:45.724998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:45.724998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:45.725169 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:45.725185 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:45.725192 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:45.725239 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:45.725301 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 22:44:45.725328 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260928 22:44:45.728221 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33337e6c8d38448c9ece3bd38252931d: Generated new TSK 0
I20260928 22:44:45.728396 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:44:45.734002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3299
2026-09-28T22:44:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1221 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3400](info): set up 2 sockets
Sep 28 22:44:45 dist-test-slave-pxr1 krb5kdc[3400](info): commencing operation
krb5kdc: starting...
W20260928 22:44:46.676437 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:46 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:46.788059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:46.866696 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:46.866937 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:46.866969 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:46.868211 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:46.868263 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:46.868287 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:46.868304 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:46.868319 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:46.869956 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:46.870221 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:46.870447 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:46.875214 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:46.875416 3413 server_base.cc:1034] running on GCE node
W20260928 22:44:46.875208 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:46.875217 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:46.875779 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:46.876097 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:46.877303 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790635486877291 us; error 32 us; skew 500 ppm
Sep 28 22:44:46 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:46.883224 3413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:46.883709 3413 webserver.cc:460] Webserver started at http://127.0.98.62:46467/ using document root <none> and password file <none>
I20260928 22:44:46.883925 3413 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:46.883975 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:46.884102 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:46.885221 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "4c387a3b7399461395a86eb06b83dfdc"
format_stamp: "Formatted at 2026-09-28 22:44:46 on dist-test-slave-pxr1"
server_key: "9fbf036e19f27eb97ee7452534dbbc9c"
server_key_iv: "3a6ec5363630693b3efe8209382cdcbf"
server_key_version: "encryptionkey@0"
I20260928 22:44:46.885561 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c387a3b7399461395a86eb06b83dfdc"
format_stamp: "Formatted at 2026-09-28 22:44:46 on dist-test-slave-pxr1"
server_key: "9fbf036e19f27eb97ee7452534dbbc9c"
server_key_iv: "3a6ec5363630693b3efe8209382cdcbf"
server_key_version: "encryptionkey@0"
I20260928 22:44:46.886854 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:46.887543 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:46.887701 3413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:46.887796 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "4c387a3b7399461395a86eb06b83dfdc"
format_stamp: "Formatted at 2026-09-28 22:44:46 on dist-test-slave-pxr1"
server_key: "9fbf036e19f27eb97ee7452534dbbc9c"
server_key_iv: "3a6ec5363630693b3efe8209382cdcbf"
server_key_version: "encryptionkey@0"
I20260928 22:44:46.887857 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:46.904886 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:46.907127 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:46.907269 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:46.911113 3413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36137
I20260928 22:44:46.911154 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36137 every 8 connection(s)
I20260928 22:44:46.911506 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:46.912523 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:46.912771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3413
I20260928 22:44:46.912854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:46.913014 392 external_mini_cluster.cc:1428] Setting key b595294433d8549354cd6f0f1ef196b6
I20260928 22:44:46.915640 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc: Bootstrap starting.
I20260928 22:44:46.916240 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:46.916487 3481 log.cc:824] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:46.917145 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc: No bootstrap required, opened a new log
Sep 28 22:44:46 dist-test-slave-pxr1 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:46.919466 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } }
I20260928 22:44:46.919612 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:46.919644 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c387a3b7399461395a86eb06b83dfdc, State: Initialized, Role: FOLLOWER
I20260928 22:44:46.919790 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } }
I20260928 22:44:46.919895 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:46.919938 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:46.919988 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:46.920681 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } }
I20260928 22:44:46.920817 3481 leader_election.cc:302] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c387a3b7399461395a86eb06b83dfdc; no voters:
I20260928 22:44:46.920996 3481 leader_election.cc:288] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:46.921037 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:46.921176 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [term 1 LEADER]: Becoming Leader. State: Replica: 4c387a3b7399461395a86eb06b83dfdc, State: Running, Role: LEADER
I20260928 22:44:46.921252 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:46.921301 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } }
I20260928 22:44:46.921838 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c387a3b7399461395a86eb06b83dfdc. Latest consensus state: current_term: 1 leader_uuid: "4c387a3b7399461395a86eb06b83dfdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } } }
I20260928 22:44:46.921916 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:46.922318 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c387a3b7399461395a86eb06b83dfdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c387a3b7399461395a86eb06b83dfdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36137 } } }
I20260928 22:44:46.922426 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:46.922544 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:46.923215 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:46.923705 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:46.913566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.62:36137)
0928 22:44:46.913746 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:46.913755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:46.913782 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:44:46.914214 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:46.914221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:46.914273 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:46.914403 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:46.915022 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:46.915950 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:46.916790 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:46.917210 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:46.917305 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:46.919821 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:46.919826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:46.919826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:46.919839 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:46.921714 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:46.922116 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:46.922120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:46.922121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:46.922225 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:46.922430 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:46.922431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:46.922432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:46.922797 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:46.922925 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:46.923377 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 22:44:46.923514 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260928 22:44:46.924952 3495 catalog_manager.cc:1347] Generated new cluster ID: be8480d1fa5a40aeb3114ceff2b697b1
I20260928 22:44:46.925001 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:46.935110 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:46.935663 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:46.940831 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c387a3b7399461395a86eb06b83dfdc: Generated new TSK 0
I20260928 22:44:46.941006 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:46.970031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:47.049429 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:47.049636 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:47.049669 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:47.050839 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:47.050880 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:47.050915 3507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:47.052335 3507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:47.052569 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:47.052810 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:47.057430 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:47.057442 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:47.057595 3507 server_base.cc:1034] running on GCE node
W20260928 22:44:47.057444 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:47.057832 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:47.058091 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:47.059227 3507 hybrid_clock.cc:648] HybridClock initialized: now 1790635487059218 us; error 29 us; skew 500 ppm
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:47.061600 3507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:47.062011 3507 webserver.cc:460] Webserver started at http://127.0.98.1:35475/ using document root <none> and password file <none>
I20260928 22:44:47.062232 3507 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:47.062279 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:47.062410 3507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:47.063441 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e10debbb4564a7fa982f54acf9efd92"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "650a52c9b4fbaa6b223d7192705a07de"
server_key_iv: "91414562b513328c788ab3f6ad7c8e99"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.063763 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e10debbb4564a7fa982f54acf9efd92"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "650a52c9b4fbaa6b223d7192705a07de"
server_key_iv: "91414562b513328c788ab3f6ad7c8e99"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.065074 3507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:47.065788 3522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.065950 3507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:47.066030 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "1e10debbb4564a7fa982f54acf9efd92"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "650a52c9b4fbaa6b223d7192705a07de"
server_key_iv: "91414562b513328c788ab3f6ad7c8e99"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.066088 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:47.074589 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:47.076249 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:47.076390 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:47.076597 3507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:47.076952 3507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:47.076988 3507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.077016 3507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:47.077041 3507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.082383 3507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39971
I20260928 22:44:47.082432 3635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39971 every 8 connection(s)
I20260928 22:44:47.082715 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:47.084708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3507
I20260928 22:44:47.084820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:47.084995 392 external_mini_cluster.cc:1428] Setting key 4f2078e39ed1804108175bb85a702df4
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:44:47.092626 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.083339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35305 (local address 127.0.98.62:36137)
0928 22:44:47.083385 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:47.083387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:47.083566 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:44:47.084070 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:47.084071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:47.084083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:47.084135 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:47.085212 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.085986 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.087361 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.087674 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.087746 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.089951 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:47.089952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:47.089952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:47.089960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:47.091291 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.091711 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.091712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.091713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.091769 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.091992 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.091993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.091993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.092108 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:47.092168 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:47.092435 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 22:44:47.092532 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:47.092800 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.083312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36137 (local address 127.0.98.1:35305)
0928 22:44:47.083452 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:47.083488 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:47.083978 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:47.084202 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:47.084206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:47.084813 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:47.085051 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.085058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.086091 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.086092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.087277 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.087280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.087745 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.087746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.087846 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.088318 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:47.088328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:47.089858 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:47.091375 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.091383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.091385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.091645 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.091823 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.091824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.091824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.091937 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.092163 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:47.092170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:47.092310 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 22:44:47.092399 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:44:47.092557 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260928 22:44:47.093266 3636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36137
I20260928 22:44:47.093359 3636 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:47.093534 3636 heartbeater.cc:511] Master 127.0.98.62:36137 requested a full tablet report, sending...
I20260928 22:44:47.094034 3445 ts_manager.cc:194] Registered new tserver with Master: 1e10debbb4564a7fa982f54acf9efd92 (127.0.98.1:39971)
I20260928 22:44:47.094704 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:47.109503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:47.189886 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:47.190092 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:47.190114 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:47.191334 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:47.191380 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:47.191416 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:47.192898 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:47.193125 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:47.193342 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:47.198879 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:47.198922 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:47.198963 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:47.199144 3641 server_base.cc:1034] running on GCE node
I20260928 22:44:47.199462 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:47.199664 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:47.200800 3641 hybrid_clock.cc:648] HybridClock initialized: now 1790635487200783 us; error 31 us; skew 500 ppm
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:47.203217 3641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:47.203678 3641 webserver.cc:460] Webserver started at http://127.0.98.2:35265/ using document root <none> and password file <none>
I20260928 22:44:47.203872 3641 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:47.203912 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:47.204002 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:47.205067 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "180f8d90fa4043d89536e0ed48eadb5b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "6ad126ee1934b697b876d294a22c02c7"
server_key_iv: "b69d4f49160704a1f8d03e2b4a8d1950"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.205371 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "180f8d90fa4043d89536e0ed48eadb5b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "6ad126ee1934b697b876d294a22c02c7"
server_key_iv: "b69d4f49160704a1f8d03e2b4a8d1950"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.206583 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:47.207252 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.207391 3641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:47.207515 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "180f8d90fa4043d89536e0ed48eadb5b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "6ad126ee1934b697b876d294a22c02c7"
server_key_iv: "b69d4f49160704a1f8d03e2b4a8d1950"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.207581 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:47.225111 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:47.227182 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:47.227334 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:47.227567 3641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:47.227865 3641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:47.227897 3641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.227927 3641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:47.227950 3641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.233503 3641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45697
I20260928 22:44:47.233557 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45697 every 8 connection(s)
I20260928 22:44:47.233891 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:47.243268 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.234631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60709 (local address 127.0.98.62:36137)
0928 22:44:47.234712 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:47.234714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:47.234886 (+ 172us) server_negotiation.cc:407] Connection header received
0928 22:44:47.235365 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:47.235366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:47.235377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:47.235447 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:47.236368 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.236922 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.237964 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.238169 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.238234 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.240629 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:47.240631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:47.240631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:47.240639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:47.241970 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.242353 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.242355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.242355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.242416 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.242643 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.242644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.242645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.242761 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:47.242817 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:47.243086 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 22:44:47.243174 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 22:44:47.243456 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.234594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36137 (local address 127.0.98.2:60709)
0928 22:44:47.234737 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:47.234780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:47.235264 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:47.235504 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:47.235508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:47.236054 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:47.236260 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.236265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.237029 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.237030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.237872 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.237874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.238268 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.238269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.238453 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.239046 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:47.239058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:47.240534 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:47.242050 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.242059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.242061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.242278 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.242471 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.242472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.242472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.242587 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.242816 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:47.242824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:47.242969 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:44:47.243059 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:44:47.243227 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 22:44:47.243934 3770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36137
I20260928 22:44:47.244027 3770 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:47.244027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3641
I20260928 22:44:47.244136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:47.244215 3770 heartbeater.cc:511] Master 127.0.98.62:36137 requested a full tablet report, sending...
I20260928 22:44:47.244325 392 external_mini_cluster.cc:1428] Setting key 40fb0cc4331e9cbd925cf8be880628ed
I20260928 22:44:47.244697 3444 ts_manager.cc:194] Registered new tserver with Master: 180f8d90fa4043d89536e0ed48eadb5b (127.0.98.2:45697)
I20260928 22:44:47.245193 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60709
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:47.269847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:47.352110 3775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:47.352339 3775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:47.352366 3775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:47.353716 3775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:47.353772 3775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:47.353801 3775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:47.355350 3775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:47.355585 3775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:47.355799 3775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:47.360536 3781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:47.360541 3780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:47.360541 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:47.360898 3775 server_base.cc:1034] running on GCE node
I20260928 22:44:47.361124 3775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:47.361358 3775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:47.362514 3775 hybrid_clock.cc:648] HybridClock initialized: now 1790635487362501 us; error 30 us; skew 500 ppm
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:47.365114 3775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:47.365574 3775 webserver.cc:460] Webserver started at http://127.0.98.3:44737/ using document root <none> and password file <none>
I20260928 22:44:47.365789 3775 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:47.365834 3775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:47.365950 3775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:47.367017 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf0c3c90ab2e4da9984fd5efc7ab908b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "bb2db44cd4cfcf5dc5b4b4d080f82691"
server_key_iv: "033b310ee359e08a0ac58690918f67c6"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.367343 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf0c3c90ab2e4da9984fd5efc7ab908b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "bb2db44cd4cfcf5dc5b4b4d080f82691"
server_key_iv: "033b310ee359e08a0ac58690918f67c6"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.368559 3775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:47.369331 3790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.369472 3775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.369544 3775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "bf0c3c90ab2e4da9984fd5efc7ab908b"
format_stamp: "Formatted at 2026-09-28 22:44:47 on dist-test-slave-pxr1"
server_key: "bb2db44cd4cfcf5dc5b4b4d080f82691"
server_key_iv: "033b310ee359e08a0ac58690918f67c6"
server_key_version: "encryptionkey@0"
I20260928 22:44:47.369608 3775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:47.386080 3775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:47.388250 3775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:47.388381 3775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:47.388609 3775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:47.388928 3775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:47.388960 3775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.388993 3775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:47.389015 3775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:47.394544 3775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39637
I20260928 22:44:47.394585 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39637 every 8 connection(s)
I20260928 22:44:47.394896 3775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:47.395190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3775
I20260928 22:44:47.395282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:47.395485 392 external_mini_cluster.cc:1428] Setting key 91079e66fee5e577ef9e9efaaad20cbb
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:47.403568 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.395677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50503 (local address 127.0.98.62:36137)
0928 22:44:47.395733 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:47.395736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:47.395935 (+ 199us) server_negotiation.cc:407] Connection header received
0928 22:44:47.396476 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:47.396477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:47.396486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:47.396531 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:47.397485 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.397976 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.398934 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.399126 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.399172 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.401221 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:47.401223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:47.401223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:47.401229 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:47.402501 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.402813 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.402814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.402814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.402862 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.403050 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.403050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.403050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.403157 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:47.403217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:47.403442 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 22:44:47.403502 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:44:47.403750 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.395667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36137 (local address 127.0.98.3:50503)
0928 22:44:47.395820 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:47.395862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:47.396376 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:47.396592 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:47.396595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:47.397202 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:47.397398 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.397413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.398045 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.398046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.398871 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:47.398874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.399185 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.399185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:47.399286 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.399764 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:47.399772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:47.401135 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:47.402565 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.402568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.402570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.402763 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.402917 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:47.402918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:47.402918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:47.403009 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:47.403207 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:47.403211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:47.403330 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 22:44:47.403413 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:44:47.403538 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 22:44:47.404219 3904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36137
I20260928 22:44:47.404304 3904 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:47.404482 3904 heartbeater.cc:511] Master 127.0.98.62:36137 requested a full tablet report, sending...
I20260928 22:44:47.404882 3444 ts_manager.cc:194] Registered new tserver with Master: bf0c3c90ab2e4da9984fd5efc7ab908b (127.0.98.3:39637)
I20260928 22:44:47.405241 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50503
I20260928 22:44:47.407086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635487, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:47.428668 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:47.422015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.0.98.62:36137)
0928 22:44:47.422069 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:47.422071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:47.422180 (+ 109us) server_negotiation.cc:407] Connection header received
0928 22:44:47.422252 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:47.422253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:47.422263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:47.422312 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:47.423003 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.423599 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.424305 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:47.424496 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:47.424550 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:47.426705 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:47.426706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:47.426707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:47.426713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:47.427794 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.428075 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.428076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.428077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.428122 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:47.428246 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:47.428246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:47.428246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:47.428378 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:47.428468 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:47.428541 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:44:47.428602 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 22:44:47.428903 3444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48530
I20260928 22:44:47.429311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3507
I20260928 22:44:47.433924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3641
I20260928 22:44:47.437893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3775
I20260928 22:44:47.441536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3413
2026-09-28T22:44:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1709 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3916](info): set up 2 sockets
Sep 28 22:44:47 dist-test-slave-pxr1 krb5kdc[3916](info): commencing operation
krb5kdc: starting...
W20260928 22:44:48.396555 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:48.512292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:48.590669 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:48.590880 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:48.590910 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:48.592080 3929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:48.592123 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:48.592140 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:48.592155 3929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:48.592170 3929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:48.593715 3929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:48.593973 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:48.594195 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:48.598865 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:48.598897 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:48.598877 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:48.599051 3929 server_base.cc:1034] running on GCE node
I20260928 22:44:48.599323 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:48.599617 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:48.600795 3929 hybrid_clock.cc:648] HybridClock initialized: now 1790635488600775 us; error 40 us; skew 500 ppm
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:48.603304 3929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:48.603746 3929 webserver.cc:460] Webserver started at http://127.0.98.62:43969/ using document root <none> and password file <none>
I20260928 22:44:48.603965 3929 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:48.604012 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:48.604117 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:48.605225 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "ca32c82b5d46491a82633c1f5ef8720b"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "d8597deb3858a88aa267229ee95d09ae"
server_key_iv: "93e50f233de2ccf0c0783fcd6530343b"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.605561 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca32c82b5d46491a82633c1f5ef8720b"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "d8597deb3858a88aa267229ee95d09ae"
server_key_iv: "93e50f233de2ccf0c0783fcd6530343b"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.606797 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:48.607492 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:48.607654 3929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:48.607728 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "ca32c82b5d46491a82633c1f5ef8720b"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "d8597deb3858a88aa267229ee95d09ae"
server_key_iv: "93e50f233de2ccf0c0783fcd6530343b"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.607797 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:48.619310 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:48.621162 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:48.621310 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:48.624841 3929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44137
I20260928 22:44:48.624886 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44137 every 8 connection(s)
I20260928 22:44:48.625203 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:48.626047 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:48.626638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 3929
I20260928 22:44:48.626720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:48.626863 392 external_mini_cluster.cc:1428] Setting key f27357c1127282a0884d08b4c3772384
I20260928 22:44:48.628602 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b: Bootstrap starting.
I20260928 22:44:48.629304 3997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:48.629590 3997 log.cc:824] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:48.630298 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b: No bootstrap required, opened a new log
I20260928 22:44:48.632443 3997 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } }
I20260928 22:44:48.632583 3997 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:48.632615 3997 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca32c82b5d46491a82633c1f5ef8720b, State: Initialized, Role: FOLLOWER
I20260928 22:44:48.632774 3997 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } }
I20260928 22:44:48.632846 3997 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:48.632876 3997 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:48.632943 3997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 0 FOLLOWER]: Advancing to term 1
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:48.633626 3997 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } }
I20260928 22:44:48.633737 3997 leader_election.cc:302] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca32c82b5d46491a82633c1f5ef8720b; no voters:
I20260928 22:44:48.633885 3997 leader_election.cc:288] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:48.633985 4002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:48.634145 3997 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:48.634167 4002 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [term 1 LEADER]: Becoming Leader. State: Replica: ca32c82b5d46491a82633c1f5ef8720b, State: Running, Role: LEADER
I20260928 22:44:48.634343 4002 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } }
I20260928 22:44:48.634690 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca32c82b5d46491a82633c1f5ef8720b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } } }
I20260928 22:44:48.634778 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:48.634984 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca32c82b5d46491a82633c1f5ef8720b. Latest consensus state: current_term: 1 leader_uuid: "ca32c82b5d46491a82633c1f5ef8720b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca32c82b5d46491a82633c1f5ef8720b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44137 } } }
I20260928 22:44:48.635056 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:48.635170 4010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:48.635634 4010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:48.637465 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:48.627356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.0.98.62:44137)
0928 22:44:48.627526 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:48.627533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:48.627559 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:44:48.627972 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:48.627979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:48.628026 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:48.628150 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:48.628781 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.630091 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.630853 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.631494 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.631613 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:48.633990 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:48.633995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:48.633995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:48.634007 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:48.636001 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:48.636357 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:48.636360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:48.636360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:48.636446 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:48.636657 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:48.636658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:48.636658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:48.636960 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:48.637109 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:48.637222 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:44:48.637323 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 22:44:48.637984 4010 catalog_manager.cc:1347] Generated new cluster ID: 5b40eb40d4784ef38b53744ec1bc70df
I20260928 22:44:48.638036 4010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:48.672073 4010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:48.672623 4010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:48.676951 4010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca32c82b5d46491a82633c1f5ef8720b: Generated new TSK 0
I20260928 22:44:48.677131 4010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:48.705999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:48.815102 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:48.815321 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:48.815344 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:48.816537 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:48.816591 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:48.816617 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:48.818207 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:48.818457 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:48.818681 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:48.823527 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:48.823691 4023 server_base.cc:1034] running on GCE node
W20260928 22:44:48.823523 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:48.823523 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:48.824005 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:48.824239 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:48.825387 4023 hybrid_clock.cc:648] HybridClock initialized: now 1790635488825373 us; error 29 us; skew 500 ppm
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:48.827852 4023 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:48.828292 4023 webserver.cc:460] Webserver started at http://127.0.98.1:40361/ using document root <none> and password file <none>
I20260928 22:44:48.828516 4023 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:48.828564 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:48.828691 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:48.829737 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "e3bba03cbf6f4a8cb9218c6d11c61635"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "4ab3bdd98707347d4302e8f7b58e3313"
server_key_iv: "9793233fb2f27c631f225f7a96a5838d"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.830068 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e3bba03cbf6f4a8cb9218c6d11c61635"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "4ab3bdd98707347d4302e8f7b58e3313"
server_key_iv: "9793233fb2f27c631f225f7a96a5838d"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.831429 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:48.832167 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:48.832342 4023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:48.832427 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "e3bba03cbf6f4a8cb9218c6d11c61635"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "4ab3bdd98707347d4302e8f7b58e3313"
server_key_iv: "9793233fb2f27c631f225f7a96a5838d"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.832492 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:48.856417 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:48.858628 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:48.858771 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:48.858989 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:48.859306 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:48.859339 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:48.859367 4023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:48.859385 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:48.864796 4023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45143
I20260928 22:44:48.864866 4151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45143 every 8 connection(s)
I20260928 22:44:48.865160 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:48.871554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4023
I20260928 22:44:48.871661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:48.871846 392 external_mini_cluster.cc:1428] Setting key 609997f3ad2d1e576928c2dd9fa41939
I20260928 22:44:48.875075 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:48.865916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46081 (local address 127.0.98.62:44137)
0928 22:44:48.865981 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:48.865983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:48.866319 (+ 336us) server_negotiation.cc:407] Connection header received
0928 22:44:48.867216 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:48.867217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:48.867229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:48.867288 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:48.867933 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.868455 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.869506 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.869713 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.869794 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:48.872085 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:48.872087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:48.872087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:48.872095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:48.873514 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:48.874007 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:48.874008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:48.874008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:48.874080 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:48.874352 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:48.874354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:48.874354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:48.874505 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:48.874606 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:48.874896 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:44:48.874985 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:44:48.875329 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:48.865868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44137 (local address 127.0.98.1:46081)
0928 22:44:48.866117 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:48.866194 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:48.867089 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:48.867360 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:48.867364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:48.867631 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:48.867828 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:48.867832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.868551 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.868552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:48.869416 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:48.869418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:48.869848 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:48.869849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:48.869972 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:48.870491 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:48.870500 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:48.872005 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:48.873595 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:48.873604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:48.873607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:48.873928 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:48.874150 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:48.874151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:48.874151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:48.874283 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:48.874623 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:48.874629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:48.874759 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 22:44:48.874859 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:44:48.875070 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260928 22:44:48.875825 4152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44137
I20260928 22:44:48.875917 4152 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:48.876097 4152 heartbeater.cc:511] Master 127.0.98.62:44137 requested a full tablet report, sending...
I20260928 22:44:48.876574 3961 ts_manager.cc:194] Registered new tserver with Master: e3bba03cbf6f4a8cb9218c6d11c61635 (127.0.98.1:45143)
I20260928 22:44:48.877189 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46081
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:48.895241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:48.973881 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:48.974095 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:48.974126 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:48.975353 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:48.975395 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:48.975430 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:48.976945 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:48.977169 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:48.977375 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:48.981973 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:48.981952 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:48.982028 4157 server_base.cc:1034] running on GCE node
W20260928 22:44:48.981952 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:48.982362 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:48.982582 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:48.983731 4157 hybrid_clock.cc:648] HybridClock initialized: now 1790635488983718 us; error 31 us; skew 500 ppm
Sep 28 22:44:48 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:48.986052 4157 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:48.986482 4157 webserver.cc:460] Webserver started at http://127.0.98.2:39527/ using document root <none> and password file <none>
I20260928 22:44:48.986666 4157 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:48.986709 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:48.986824 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:48.987819 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "de260b53059042a0811593b66d047539"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "04ea2a4883fc0e3a6babf152026d7247"
server_key_iv: "f7c875d4f0641738e2b11c92ab7b5da4"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.988123 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de260b53059042a0811593b66d047539"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "04ea2a4883fc0e3a6babf152026d7247"
server_key_iv: "f7c875d4f0641738e2b11c92ab7b5da4"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.989390 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:48.990016 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:48.990165 4157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:48.990252 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "de260b53059042a0811593b66d047539"
format_stamp: "Formatted at 2026-09-28 22:44:48 on dist-test-slave-pxr1"
server_key: "04ea2a4883fc0e3a6babf152026d7247"
server_key_iv: "f7c875d4f0641738e2b11c92ab7b5da4"
server_key_version: "encryptionkey@0"
I20260928 22:44:48.990309 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:49.005846 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:49.007707 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:49.007844 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:49.008060 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:49.008358 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:49.008391 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:49.008420 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:49.008437 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:49.013948 4157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45293
I20260928 22:44:49.014034 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45293 every 8 connection(s)
I20260928 22:44:49.014322 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:49.019532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4157
I20260928 22:44:49.019637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:49.019784 392 external_mini_cluster.cc:1428] Setting key 2ec00062a9d624104181db782847586d
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:49.023447 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.015016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52909 (local address 127.0.98.62:44137)
0928 22:44:49.015079 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:49.015081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:49.015281 (+ 200us) server_negotiation.cc:407] Connection header received
0928 22:44:49.016195 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:49.016196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:49.016205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:49.016257 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:49.016906 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.017411 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.018403 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.018580 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.018627 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.020706 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:49.020707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:49.020707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:49.020734 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:49.022105 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.022564 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.022564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.022565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.022611 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.022863 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.022863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.022864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.022961 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:49.023008 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:49.023314 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 22:44:49.023383 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:44:49.023748 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.014974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44137 (local address 127.0.98.2:52909)
0928 22:44:49.015159 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:49.015196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:49.016090 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:49.016316 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:49.016319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:49.016570 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:49.016826 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:49.016831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.017497 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.017498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:49.018337 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:49.018339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.018643 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.018643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:49.018740 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.019180 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:49.019189 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:49.020612 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:49.022189 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:49.022197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:49.022200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:49.022497 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:49.022667 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:49.022668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:49.022668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:49.022800 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:49.023007 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:49.023013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:49.023187 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:44:49.023302 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 22:44:49.023468 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260928 22:44:49.024406 4286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44137
I20260928 22:44:49.024509 4286 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:49.024684 4286 heartbeater.cc:511] Master 127.0.98.62:44137 requested a full tablet report, sending...
I20260928 22:44:49.025094 3961 ts_manager.cc:194] Registered new tserver with Master: de260b53059042a0811593b66d047539 (127.0.98.2:45293)
I20260928 22:44:49.025463 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52909
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:49.043887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:49.122732 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:49.122953 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:49.122977 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:49.124229 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:49.124270 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:49.124300 4291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:49.125837 4291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:49.126063 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:49.126305 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:49.131388 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:49.131462 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:49.131388 4299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:49.131464 4291 server_base.cc:1034] running on GCE node
I20260928 22:44:49.131814 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:49.132081 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:49.133225 4291 hybrid_clock.cc:648] HybridClock initialized: now 1790635489133208 us; error 37 us; skew 500 ppm
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:49.136207 4291 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:49.136636 4291 webserver.cc:460] Webserver started at http://127.0.98.3:44763/ using document root <none> and password file <none>
I20260928 22:44:49.136894 4291 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:49.136946 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:49.137079 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:49.138131 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9e704cb97bf4c60a83b47b1be435331"
format_stamp: "Formatted at 2026-09-28 22:44:49 on dist-test-slave-pxr1"
server_key: "46fde0618b58b83a692a7c654810f135"
server_key_iv: "482ccfd13acfe91912d1273c2c4ca642"
server_key_version: "encryptionkey@0"
I20260928 22:44:49.138484 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9e704cb97bf4c60a83b47b1be435331"
format_stamp: "Formatted at 2026-09-28 22:44:49 on dist-test-slave-pxr1"
server_key: "46fde0618b58b83a692a7c654810f135"
server_key_iv: "482ccfd13acfe91912d1273c2c4ca642"
server_key_version: "encryptionkey@0"
I20260928 22:44:49.139707 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:49.140448 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:49.140612 4291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:49.140699 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "c9e704cb97bf4c60a83b47b1be435331"
format_stamp: "Formatted at 2026-09-28 22:44:49 on dist-test-slave-pxr1"
server_key: "46fde0618b58b83a692a7c654810f135"
server_key_iv: "482ccfd13acfe91912d1273c2c4ca642"
server_key_version: "encryptionkey@0"
I20260928 22:44:49.140797 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:49.159175 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:49.161206 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:49.161337 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:49.161548 4291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:49.161837 4291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:49.161870 4291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:49.161902 4291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:49.161926 4291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:49.167315 4291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35097
I20260928 22:44:49.167356 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35097 every 8 connection(s)
I20260928 22:44:49.167657 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:49.168357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4291
I20260928 22:44:49.168455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:49.168635 392 external_mini_cluster.cc:1428] Setting key 6cd7ca4ba17292104300564f623adb1f
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:49.177040 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.168342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59211 (local address 127.0.98.62:44137)
0928 22:44:49.168390 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:49.168392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:49.168938 (+ 546us) server_negotiation.cc:407] Connection header received
0928 22:44:49.169955 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:49.169956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:49.169965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:49.170006 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:49.170568 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.171055 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.171977 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.172150 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.172198 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.174210 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:49.174212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:49.174212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:49.174220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:49.175601 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.175960 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.175961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.175961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.176006 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.176232 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.176234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.176234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.176383 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:49.176452 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:49.176852 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 22:44:49.176944 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:44:49.177233 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.168339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44137 (local address 127.0.98.3:59211)
0928 22:44:49.168803 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:49.168842 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:49.169874 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:49.170066 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:49.170070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:49.170318 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:49.170509 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:49.170513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.171115 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.171116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:49.171918 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:49.171920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.172208 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.172208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:49.172308 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.172776 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:49.172786 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:49.174108 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:49.175688 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:49.175693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:49.175695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:49.175901 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:49.176050 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:49.176051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:49.176051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:49.176140 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:49.176447 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:49.176454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:49.176662 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 22:44:49.176801 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 22:44:49.176998 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":83,"threads_started":1}
I20260928 22:44:49.177795 4420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44137
I20260928 22:44:49.177886 4420 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:49.178056 4420 heartbeater.cc:511] Master 127.0.98.62:44137 requested a full tablet report, sending...
I20260928 22:44:49.178390 3961 ts_manager.cc:194] Registered new tserver with Master: c9e704cb97bf4c60a83b47b1be435331 (127.0.98.3:35097)
I20260928 22:44:49.178831 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59211
I20260928 22:44:49.180176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:49.202028 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.194302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.0.98.1:45143)
0928 22:44:49.194454 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:49.194460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:49.194470 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:49.194496 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:49.194498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:49.194512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:49.194589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:49.195186 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.195883 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.196586 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.196936 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.196989 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.199106 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:49.199115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:49.199115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:49.199123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:49.200573 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.201019 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.201023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.201023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.201114 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.201380 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.201381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.201381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.201579 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:49.201738 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:49.201828 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:44:49.201916 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:49.210386 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:49.204386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.62:44137)
0928 22:44:49.204424 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:49.204426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:49.204483 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:44:49.204534 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:49.204535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:49.204543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:49.204585 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:49.205068 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.205564 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.206191 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:49.206371 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:49.206414 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:49.208368 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:49.208369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:49.208369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:49.208376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:49.209502 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.209773 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.209773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.209774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.209818 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:49.209934 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:49.209935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:49.209935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:49.210062 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:49.210152 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:49.210217 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:44:49.210299 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:44:49.210575 3961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252
I20260928 22:44:49.210866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4023
I20260928 22:44:49.215118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4157
I20260928 22:44:49.218924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4291
I20260928 22:44:49.222658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 3929
2026-09-28T22:44:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1781 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[4432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[4432](info): set up 2 sockets
Sep 28 22:44:49 dist-test-slave-pxr1 krb5kdc[4432](info): commencing operation
krb5kdc: starting...
W20260928 22:44:50.168375 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:50.279040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44691
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:50.359210 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:50.359421 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:50.359450 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:50.360608 4445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:50.360648 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:50.360666 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:50.360682 4445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:50.360698 4445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:50.362244 4445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:50.362478 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:50.362689 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:50.367151 4450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.367223 4453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.367226 4451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.367304 4445 server_base.cc:1034] running on GCE node
I20260928 22:44:50.367607 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:50.367882 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:50.369041 4445 hybrid_clock.cc:648] HybridClock initialized: now 1790635490369025 us; error 32 us; skew 500 ppm
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:50.371351 4445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:50.371672 4445 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:50.371730 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:50.371855 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:50.372913 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "b272766ebafe4b599ed1007250028b2d"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "c1feebc42fdbbb3a69b2c77b20db82c6"
server_key_iv: "adde62b37149cb53d1aa0d9ae1e764c8"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.373234 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "b272766ebafe4b599ed1007250028b2d"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "c1feebc42fdbbb3a69b2c77b20db82c6"
server_key_iv: "adde62b37149cb53d1aa0d9ae1e764c8"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.374385 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:50.375033 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.375178 4445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:50.375262 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "b272766ebafe4b599ed1007250028b2d"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "c1feebc42fdbbb3a69b2c77b20db82c6"
server_key_iv: "adde62b37149cb53d1aa0d9ae1e764c8"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.375319 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:50.384184 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:50.386230 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:50.386361 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:50.390004 4445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44691
I20260928 22:44:50.390036 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44691 every 8 connection(s)
I20260928 22:44:50.390317 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:50.391161 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:50.393141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4445
I20260928 22:44:50.393232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:50.393405 392 external_mini_cluster.cc:1428] Setting key ebd4c1ee05f191104398ed510af1a8ec
I20260928 22:44:50.393519 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: Bootstrap starting.
I20260928 22:44:50.394171 4512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:50.394435 4512 log.cc:824] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:50.395151 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: No bootstrap required, opened a new log
I20260928 22:44:50.397065 4512 raft_consensus.cc:348] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } }
I20260928 22:44:50.397230 4512 raft_consensus.cc:374] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:50.397313 4512 raft_consensus.cc:729] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b272766ebafe4b599ed1007250028b2d, State: Initialized, Role: FOLLOWER
I20260928 22:44:50.397441 4512 consensus_queue.cc:260] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } }
I20260928 22:44:50.397516 4512 raft_consensus.cc:388] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:50.397553 4512 raft_consensus.cc:482] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:50.397599 4512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:50.398156 4512 raft_consensus.cc:504] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } }
I20260928 22:44:50.398255 4512 leader_election.cc:302] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b272766ebafe4b599ed1007250028b2d; no voters:
I20260928 22:44:50.398401 4512 leader_election.cc:288] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:50.398451 4517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:50.398630 4517 raft_consensus.cc:686] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [term 1 LEADER]: Becoming Leader. State: Replica: b272766ebafe4b599ed1007250028b2d, State: Running, Role: LEADER
I20260928 22:44:50.398651 4512 sys_catalog.cc:564] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:50.398779 4517 consensus_queue.cc:237] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } }
I20260928 22:44:50.399147 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b272766ebafe4b599ed1007250028b2d. Latest consensus state: current_term: 1 leader_uuid: "b272766ebafe4b599ed1007250028b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } } }
I20260928 22:44:50.399219 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:50.399430 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b272766ebafe4b599ed1007250028b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272766ebafe4b599ed1007250028b2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44691 } } }
I20260928 22:44:50.399488 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:50.399818 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:50.400301 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:50.401948 4523 catalog_manager.cc:1347] Generated new cluster ID: 39f9441575d74535b999f6d2ea59af56
I20260928 22:44:50.402002 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:50.403636 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.393868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.0.98.62:44691)
0928 22:44:50.394020 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.394030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.394056 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:44:50.394494 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.394503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.394562 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.394707 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:50.395366 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.396220 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.397294 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.397818 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.397906 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.400553 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:50.400557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:50.400558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:50.400570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:50.402298 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.402604 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.402607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.402607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.402701 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.402870 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.402871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.402872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.403100 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:50.403238 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.403341 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.403465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260928 22:44:50.407752 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:50.408231 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:50.413344 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: Generated new TSK 0
I20260928 22:44:50.413511 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:50.439419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:50.541096 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:50.541297 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:50.541328 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:50.542490 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:50.542548 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:50.542579 4538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:50.544150 4538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:50.544368 4538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:50.544576 4538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:50.549273 4543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.549288 4544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.549415 4538 server_base.cc:1034] running on GCE node
W20260928 22:44:50.549304 4546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.549635 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:50.549861 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:50.551002 4538 hybrid_clock.cc:648] HybridClock initialized: now 1790635490550980 us; error 44 us; skew 500 ppm
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:50.553431 4538 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:50.553750 4538 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:50.553802 4538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:50.553922 4538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:50.554992 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "f22d781af7ed4c27aa21ebcf17c9353e"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "acdd6f60fe882c6430ef4b94f6a59a4d"
server_key_iv: "c0b0531c44dd12d291152b8230ce776c"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.555341 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f22d781af7ed4c27aa21ebcf17c9353e"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "acdd6f60fe882c6430ef4b94f6a59a4d"
server_key_iv: "c0b0531c44dd12d291152b8230ce776c"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.556564 4538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:50.557293 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.557438 4538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.557511 4538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "f22d781af7ed4c27aa21ebcf17c9353e"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "acdd6f60fe882c6430ef4b94f6a59a4d"
server_key_iv: "c0b0531c44dd12d291152b8230ce776c"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.557578 4538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:50.583153 4538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:50.585421 4538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:50.585566 4538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:50.585798 4538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:50.586113 4538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:50.586149 4538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.586179 4538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:50.586202 4538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.591343 4538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42813
I20260928 22:44:50.591382 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42813 every 8 connection(s)
I20260928 22:44:50.591634 4538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:50.594699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4538
I20260928 22:44:50.594795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:50.594945 392 external_mini_cluster.cc:1428] Setting key 86f7454ad4a2064e1ac561bedc8fb067
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:50.601177 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.592317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41721 (local address 127.0.98.62:44691)
0928 22:44:50.592374 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.592376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.592570 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:44:50.593083 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.593084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.593095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.593142 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:50.594022 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.594521 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.595710 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.595923 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.595977 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.598434 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:50.598436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:50.598436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:50.598444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:50.599773 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.600262 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.600263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.600264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.600319 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.600568 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.600569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.600569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.600668 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:50.600740 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.601050 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.601114 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:44:50.601446 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.592296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44691 (local address 127.0.98.1:41721)
0928 22:44:50.592457 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.592493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.593004 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.593208 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.593211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.593462 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:50.593952 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.593957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.594630 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.594631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.595637 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.595640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.595995 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.595996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.596131 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.596645 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:50.596655 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:50.598321 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:50.599845 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.599853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.599856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.600180 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.600375 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.600376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.600377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.600512 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.600742 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:50.600748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:50.600920 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.601040 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.601222 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 22:44:50.601962 4666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44691
I20260928 22:44:50.602074 4666 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:50.602224 4666 heartbeater.cc:511] Master 127.0.98.62:44691 requested a full tablet report, sending...
I20260928 22:44:50.602708 4476 ts_manager.cc:194] Registered new tserver with Master: f22d781af7ed4c27aa21ebcf17c9353e (127.0.98.1:42813)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:44:50.603310 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:50.619761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:50.698297 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:50.698499 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:50.698530 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:50.699710 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:50.699754 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:50.699790 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:50.701340 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:50.701578 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:50.701787 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:50.706492 4676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.706503 4679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.706553 4677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.706638 4671 server_base.cc:1034] running on GCE node
I20260928 22:44:50.706919 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:50.707131 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:50.708312 4671 hybrid_clock.cc:648] HybridClock initialized: now 1790635490708295 us; error 31 us; skew 500 ppm
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:50.710762 4671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:50.711088 4671 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:50.711138 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:50.711241 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:50.712240 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "99913c53c6b9475e9998be7a787769ec"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "8e4be756df0cb5c2b3a5a49c0c429d76"
server_key_iv: "a74e17477e3fe510f181366eb74ac965"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.712533 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "99913c53c6b9475e9998be7a787769ec"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "8e4be756df0cb5c2b3a5a49c0c429d76"
server_key_iv: "a74e17477e3fe510f181366eb74ac965"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.713761 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:50.714396 4685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.714529 4671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:50.714612 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "99913c53c6b9475e9998be7a787769ec"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "8e4be756df0cb5c2b3a5a49c0c429d76"
server_key_iv: "a74e17477e3fe510f181366eb74ac965"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.714672 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:50.729247 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:50.731460 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:50.731599 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:50.731817 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:50.732112 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:50.732144 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.732182 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:50.732206 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.737661 4671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45739
I20260928 22:44:50.737713 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45739 every 8 connection(s)
I20260928 22:44:50.737964 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:50.744117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4671
I20260928 22:44:50.744228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:50.744407 392 external_mini_cluster.cc:1428] Setting key a461cd7cf5269fe8998f8eb62668b75c
I20260928 22:44:50.746901 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.738648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51567 (local address 127.0.98.62:44691)
0928 22:44:50.738703 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.738705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.738915 (+ 210us) server_negotiation.cc:407] Connection header received
0928 22:44:50.739386 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.739387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.739397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.739460 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:50.740389 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.740913 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.741859 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.742040 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.742083 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.744090 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:50.744092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:50.744093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:50.744101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:50.745502 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.745964 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.745965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.745965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.746027 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.746270 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.746271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.746271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.746400 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:50.746468 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.746731 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.746808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:44:50.747059 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.738611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44691 (local address 127.0.98.2:51567)
0928 22:44:50.738782 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.738825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.739304 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.739534 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.739537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.739807 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:50.740299 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.740305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.740983 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.740984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.741795 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.741797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.742097 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.742098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.742204 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.742693 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:50.742702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:50.744002 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:50.745599 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.745606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.745609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.745892 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.746082 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.746083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.746083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.746214 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.746465 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:50.746470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:50.746615 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.746699 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.746825 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 22:44:50.747457 4799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44691
I20260928 22:44:50.747545 4799 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:50.747692 4799 heartbeater.cc:511] Master 127.0.98.62:44691 requested a full tablet report, sending...
I20260928 22:44:50.748052 4476 ts_manager.cc:194] Registered new tserver with Master: 99913c53c6b9475e9998be7a787769ec (127.0.98.2:45739)
I20260928 22:44:50.748546 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51567
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:50.768325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:50.848605 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:50.848832 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:50.848858 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:50.850040 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:50.850080 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:50.850108 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:50.851615 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:50.851845 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:50.852064 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:50.856669 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.856823 4804 server_base.cc:1034] running on GCE node
W20260928 22:44:50.856676 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:50.856688 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:50.857126 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:50.857342 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:50.858474 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790635490858466 us; error 40 us; skew 500 ppm
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:50.860914 4804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:50.861235 4804 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:50.861285 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:50.861413 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:50.862447 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "29c4b5aecfad57b562e9e176d0be03b7"
server_key_iv: "fb6fc50919f7816813f3fb81dfefc389"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.862767 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "29c4b5aecfad57b562e9e176d0be03b7"
server_key_iv: "fb6fc50919f7816813f3fb81dfefc389"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.863935 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:50.864637 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.864811 4804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:50.864890 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a"
format_stamp: "Formatted at 2026-09-28 22:44:50 on dist-test-slave-pxr1"
server_key: "29c4b5aecfad57b562e9e176d0be03b7"
server_key_iv: "fb6fc50919f7816813f3fb81dfefc389"
server_key_version: "encryptionkey@0"
I20260928 22:44:50.864948 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:50.886432 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:50.888304 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:50.888439 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:50.888655 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:50.888999 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:50.889034 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.889063 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:50.889081 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:50.894392 4804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44621
I20260928 22:44:50.894449 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44621 every 8 connection(s)
I20260928 22:44:50.894702 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:44:50 dist-test-slave-pxr1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:50.903216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4804
I20260928 22:44:50.903350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:50.903568 392 external_mini_cluster.cc:1428] Setting key 03ee9f84e5877d9f48c3cb5cfa94299d
I20260928 22:44:50.904261 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.895509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58571 (local address 127.0.98.62:44691)
0928 22:44:50.895580 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.895582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.895805 (+ 223us) server_negotiation.cc:407] Connection header received
0928 22:44:50.896314 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.896315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.896327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.896384 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:50.897372 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.897888 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.899018 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.899269 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.899328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.901475 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:50.901476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:50.901476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:50.901492 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:50.902654 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.903087 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.903089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.903089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.903159 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.903506 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.903507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.903508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.903663 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:50.903731 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.904084 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.904178 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:44:50.904440 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.895475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44691 (local address 127.0.98.3:58571)
0928 22:44:50.895683 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.895721 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.896201 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.896453 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.896457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.896763 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:50.897270 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.897276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.897991 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.897992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.898889 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.898892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.899347 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.899348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.899468 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.899976 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:50.899985 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:50.901375 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:50.902735 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.902741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.902743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.902999 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.903242 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:50.903243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:50.903244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:50.903372 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:50.903732 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:50.903739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:50.903914 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.904049 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.904224 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260928 22:44:50.904834 4932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44691
I20260928 22:44:50.904927 4932 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:50.905076 4932 heartbeater.cc:511] Master 127.0.98.62:44691 requested a full tablet report, sending...
I20260928 22:44:50.905402 4476 ts_manager.cc:194] Registered new tserver with Master: 2ac5d35b3338424fbfd06fd3bdc1cb4a (127.0.98.3:44621)
I20260928 22:44:50.905755 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58571
I20260928 22:44:50.915184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:44:50.921639 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.916374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.0.98.62:44691)
0928 22:44:50.916428 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.916429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.916483 (+ 54us) server_negotiation.cc:407] Connection header received
0928 22:44:50.916526 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.916526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.916535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.916586 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:50.917116 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.917637 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.918325 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.918532 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.918586 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.919581 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:50.919582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:50.919583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:50.919588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:50.920812 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.921090 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.921090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.921090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.921136 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:50.921249 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:50.921250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:50.921250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:50.921372 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:50.921437 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.921516 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.921574 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:50.922969 4476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:50.924455 4476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:50.930832 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.926730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42813 (local address 127.0.0.1:40570)
0928 22:44:50.927020 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.927042 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.927104 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.927655 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.927656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.927658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:50.927698 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.927700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928540 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.928541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.929922 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.929924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930545 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.930617 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930631 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.930696 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.930748 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260928 22:44:50.931071 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.927107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.98.2:45739)
0928 22:44:50.927249 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.927254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.927307 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:44:50.927388 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.927390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.927405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.927492 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:50.927686 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928742 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.929807 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930544 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930613 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930673 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.930944 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.930985 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 22:44:50.930897 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.926891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.0.98.3:44621)
0928 22:44:50.927064 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.927069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.927079 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:50.927119 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.927121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.927137 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.927216 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:50.927356 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928598 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.929941 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930608 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930664 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930708 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.930779 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.930816 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260928 22:44:50.931123 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.926833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.0.98.1:42813)
0928 22:44:50.927433 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.927437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.927446 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:50.927481 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.927483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.927496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.927574 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:50.927767 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928452 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.929985 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930474 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930535 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930608 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.931013 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.931055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":94,"threads_started":1}
I20260928 22:44:50.930902 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.926730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44621 (local address 127.0.0.1:48198)
0928 22:44:50.926880 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.926920 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.926980 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.927217 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.927220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.927222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:50.927295 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.927297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928691 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.928691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.929865 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.929868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930665 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.930718 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930723 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.930767 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.930803 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260928 22:44:50.930979 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.926906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45739 (local address 127.0.0.1:34400)
0928 22:44:50.927254 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.927263 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.927316 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.927577 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.927578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.927580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:50.927621 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.927623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.928814 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.928815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.929748 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.929750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.930709 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.930710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.930773 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.930778 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.930822 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.930861 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":68,"threads_started":1}
I20260928 22:44:50.931947 4600 tablet_service.cc:1467] Processing CreateTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f (DEFAULT_TABLE table=test-table [id=7a114f1801144c69b9118530ddc403e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:50.931947 4866 tablet_service.cc:1467] Processing CreateTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f (DEFAULT_TABLE table=test-table [id=7a114f1801144c69b9118530ddc403e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:50.931947 4733 tablet_service.cc:1467] Processing CreateTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f (DEFAULT_TABLE table=test-table [id=7a114f1801144c69b9118530ddc403e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:50.932288 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24ee8bc41c5423da66e2d9fe1b61f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:50.932289 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24ee8bc41c5423da66e2d9fe1b61f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:50.932289 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24ee8bc41c5423da66e2d9fe1b61f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:50.935421 4951 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: Bootstrap starting.
I20260928 22:44:50.935597 4949 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Bootstrap starting.
I20260928 22:44:50.936022 4951 tablet_bootstrap.cc:654] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:50.936112 4949 tablet_bootstrap.cc:654] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:50.936491 4951 log.cc:824] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:50.936556 4949 log.cc:824] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:50.936834 4950 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Bootstrap starting.
I20260928 22:44:50.937256 4949 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: No bootstrap required, opened a new log
I20260928 22:44:50.937318 4949 ts_tablet_manager.cc:1397] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:44:50.937353 4950 tablet_bootstrap.cc:654] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:50.937803 4950 log.cc:824] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:50.938330 4951 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: No bootstrap required, opened a new log
I20260928 22:44:50.938395 4951 ts_tablet_manager.cc:1397] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:44:50.938892 4949 raft_consensus.cc:348] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.939044 4949 raft_consensus.cc:374] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:50.939078 4949 raft_consensus.cc:729] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99913c53c6b9475e9998be7a787769ec, State: Initialized, Role: FOLLOWER
I20260928 22:44:50.939193 4949 consensus_queue.cc:260] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.939445 4949 ts_tablet_manager.cc:1428] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:44:50.939460 4799 heartbeater.cc:503] Master 127.0.98.62:44691 was elected leader, sending a full tablet report...
I20260928 22:44:50.940074 4951 raft_consensus.cc:348] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.940202 4951 raft_consensus.cc:374] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:50.940236 4951 raft_consensus.cc:729] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22d781af7ed4c27aa21ebcf17c9353e, State: Initialized, Role: FOLLOWER
I20260928 22:44:50.940348 4951 consensus_queue.cc:260] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.940629 4951 ts_tablet_manager.cc:1428] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:44:50.940634 4666 heartbeater.cc:503] Master 127.0.98.62:44691 was elected leader, sending a full tablet report...
I20260928 22:44:50.941046 4950 tablet_bootstrap.cc:492] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: No bootstrap required, opened a new log
I20260928 22:44:50.941121 4950 ts_tablet_manager.cc:1397] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 22:44:50.942667 4950 raft_consensus.cc:348] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.942827 4950 raft_consensus.cc:374] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:50.942857 4950 raft_consensus.cc:729] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac5d35b3338424fbfd06fd3bdc1cb4a, State: Initialized, Role: FOLLOWER
I20260928 22:44:50.942942 4950 consensus_queue.cc:260] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.943145 4950 ts_tablet_manager.cc:1428] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 22:44:50.943183 4932 heartbeater.cc:503] Master 127.0.98.62:44691 was elected leader, sending a full tablet report...
I20260928 22:44:50.959832 4956 raft_consensus.cc:482] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:50.959955 4956 raft_consensus.cc:504] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.960340 4956 leader_election.cc:288] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac5d35b3338424fbfd06fd3bdc1cb4a (127.0.98.3:44621), 99913c53c6b9475e9998be7a787769ec (127.0.98.2:45739)
I20260928 22:44:50.963305 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.960651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43793 (local address 127.0.98.3:44621)
0928 22:44:50.960697 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.960700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.960710 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:50.960767 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.960768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.960780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.960842 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:50.961040 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.961511 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.962624 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.962995 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.963063 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.963074 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.963207 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.963241 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:44:50.963305 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.960579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44621 (local address 127.0.98.1:43793)
0928 22:44:50.960630 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.960641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.960686 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.960899 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.960900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.960902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:50.960970 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.960972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.961588 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.961589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.962541 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.962543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.963065 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.963066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.963143 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.963146 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.963192 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.963232 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:44:50.963387 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.960882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56089 (local address 127.0.98.2:45739)
0928 22:44:50.960956 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.960958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.960967 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:50.961024 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.961025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.961035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.961080 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:50.961221 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.961666 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.962710 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.963072 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.963132 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.963142 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.963279 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.963314 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:44:50.963423 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.960686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45739 (local address 127.0.98.1:56089)
0928 22:44:50.960902 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.960912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:50.960961 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:50.961088 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:50.961088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:50.961090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:50.961161 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.961163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.961732 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.961733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.962636 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:50.962639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.963152 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.963152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:50.963220 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.963224 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:50.963265 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 22:44:50.963303 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":137,"threads_started":1}
I20260928 22:44:50.963650 4886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24ee8bc41c5423da66e2d9fe1b61f6f" candidate_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" is_pre_election: true
I20260928 22:44:50.963692 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24ee8bc41c5423da66e2d9fe1b61f6f" candidate_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913c53c6b9475e9998be7a787769ec" is_pre_election: true
I20260928 22:44:50.963829 4753 raft_consensus.cc:2454] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22d781af7ed4c27aa21ebcf17c9353e in term 0.
I20260928 22:44:50.963829 4886 raft_consensus.cc:2454] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22d781af7ed4c27aa21ebcf17c9353e in term 0.
I20260928 22:44:50.964035 4553 leader_election.cc:302] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac5d35b3338424fbfd06fd3bdc1cb4a, f22d781af7ed4c27aa21ebcf17c9353e; no voters:
I20260928 22:44:50.964154 4956 raft_consensus.cc:2781] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:50.964210 4956 raft_consensus.cc:482] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:50.964231 4956 raft_consensus.cc:3037] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:50.964892 4956 raft_consensus.cc:504] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.965006 4956 leader_election.cc:288] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [CANDIDATE]: Term 1 election: Requested vote from peers 2ac5d35b3338424fbfd06fd3bdc1cb4a (127.0.98.3:44621), 99913c53c6b9475e9998be7a787769ec (127.0.98.2:45739)
I20260928 22:44:50.965175 4886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24ee8bc41c5423da66e2d9fe1b61f6f" candidate_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a"
I20260928 22:44:50.965198 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24ee8bc41c5423da66e2d9fe1b61f6f" candidate_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99913c53c6b9475e9998be7a787769ec"
I20260928 22:44:50.965262 4753 raft_consensus.cc:3037] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:50.965260 4886 raft_consensus.cc:3037] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:50.965925 4753 raft_consensus.cc:2454] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22d781af7ed4c27aa21ebcf17c9353e in term 1.
I20260928 22:44:50.965965 4886 raft_consensus.cc:2454] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22d781af7ed4c27aa21ebcf17c9353e in term 1.
I20260928 22:44:50.966136 4553 leader_election.cc:302] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac5d35b3338424fbfd06fd3bdc1cb4a, f22d781af7ed4c27aa21ebcf17c9353e; no voters:
I20260928 22:44:50.966234 4956 raft_consensus.cc:2781] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:50.966392 4956 raft_consensus.cc:686] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [term 1 LEADER]: Becoming Leader. State: Replica: f22d781af7ed4c27aa21ebcf17c9353e, State: Running, Role: LEADER
I20260928 22:44:50.966482 4956 consensus_queue.cc:237] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } }
I20260928 22:44:50.967321 4476 catalog_manager.cc:5495] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e reported cstate change: term changed from 0 to 1, leader changed from <none> to f22d781af7ed4c27aa21ebcf17c9353e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22d781af7ed4c27aa21ebcf17c9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:50.975843 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.973761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.0.98.1:42813)
0928 22:44:50.973807 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:50.973809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:50.973816 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:50.973849 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:50.973850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:50.973866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:50.973936 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:50.974062 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.974514 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.975266 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:50.975436 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:50.975475 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:50.975594 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:50.975691 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:50.975744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:50.975757 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 22:44:50.975772 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:44:50.977614 4886 raft_consensus.cc:1261] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 1 FOLLOWER]: Refusing update from remote peer f22d781af7ed4c27aa21ebcf17c9353e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:50.977615 4753 raft_consensus.cc:1261] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 1 FOLLOWER]: Refusing update from remote peer f22d781af7ed4c27aa21ebcf17c9353e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:50.978027 4959 consensus_queue.cc:1035] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99913c53c6b9475e9998be7a787769ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:50.978156 4956 consensus_queue.cc:1035] T e24ee8bc41c5423da66e2d9fe1b61f6f P f22d781af7ed4c27aa21ebcf17c9353e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac5d35b3338424fbfd06fd3bdc1cb4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:50.981000 4963 mvcc.cc:205] Tried to move back new op lower bound from 7334442971041705984 to 7334442970999480320. Current Snapshot: MvccSnapshot[applied={T|T < 7334442971041705984}]
I20260928 22:44:50.982777 4475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:50.982863 4475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:50.983562 4475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b272766ebafe4b599ed1007250028b2d: Sending DeleteTablet for 3 replicas of tablet e24ee8bc41c5423da66e2d9fe1b61f6f
I20260928 22:44:50.983915 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40570
I20260928 22:44:50.983915 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48198
I20260928 22:44:50.984148 4972 tablet_replica.cc:330] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: stopping tablet replica
I20260928 22:44:50.984150 4733 tablet_service.cc:1514] Processing DeleteTablet for tablet e24ee8bc41c5423da66e2d9fe1b61f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34400
I20260928 22:44:50.984252 4972 raft_consensus.cc:2229] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:50.984344 4974 tablet_replica.cc:330] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: stopping tablet replica
I20260928 22:44:50.984347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4538
I20260928 22:44:50.984387 4972 raft_consensus.cc:2258] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:50.984462 4974 raft_consensus.cc:2229] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:50.984570 4974 raft_consensus.cc:2258] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:50.985208 4974 ts_tablet_manager.cc:1905] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:50.985709 4972 ts_tablet_manager.cc:1905] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:50.986747 4974 ts_tablet_manager.cc:1918] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:50.986817 4974 log.cc:1196] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/e24ee8bc41c5423da66e2d9fe1b61f6f
I20260928 22:44:50.987067 4974 ts_tablet_manager.cc:1939] T e24ee8bc41c5423da66e2d9fe1b61f6f P 99913c53c6b9475e9998be7a787769ec: Deleting consensus metadata
I20260928 22:44:50.987234 4972 ts_tablet_manager.cc:1918] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:50.987299 4972 log.cc:1196] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/e24ee8bc41c5423da66e2d9fe1b61f6f
I20260928 22:44:50.987526 4460 catalog_manager.cc:4841] TS 99913c53c6b9475e9998be7a787769ec (127.0.98.2:45739): tablet e24ee8bc41c5423da66e2d9fe1b61f6f (table test-table [id=7a114f1801144c69b9118530ddc403e9]) successfully deleted
I20260928 22:44:50.987572 4972 ts_tablet_manager.cc:1939] T e24ee8bc41c5423da66e2d9fe1b61f6f P 2ac5d35b3338424fbfd06fd3bdc1cb4a: Deleting consensus metadata
I20260928 22:44:50.987906 4460 catalog_manager.cc:4841] TS 2ac5d35b3338424fbfd06fd3bdc1cb4a (127.0.98.3:44621): tablet e24ee8bc41c5423da66e2d9fe1b61f6f (table test-table [id=7a114f1801144c69b9118530ddc403e9]) successfully deleted
W20260928 22:44:50.990078 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42813 (error 108)
I20260928 22:44:50.990202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4671
I20260928 22:44:50.990993 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:50.990871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42813 (local address 127.0.0.1:40584)
0928 22:44:50.990920 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:50.990960 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:44:50.991138 4461 catalog_manager.cc:4584] TS f22d781af7ed4c27aa21ebcf17c9353e (127.0.98.1:42813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e24ee8bc41c5423da66e2d9fe1b61f6f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42813: connect: Connection refused (error 111)
I20260928 22:44:50.994743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4804
I20260928 22:44:50.998714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4445
2026-09-28T22:44:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1775 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T22:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:51Z Disabled control of system clock
I20260928 22:44:51.023761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45935
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:44:51.102596 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.102821 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.102842 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.104043 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:51.104095 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.104107 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.104118 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:51.104130 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:51.105574 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.105825 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.106132 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.110821 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.110816 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.110821 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.111090 4979 server_base.cc:1034] running on GCE node
I20260928 22:44:51.111379 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.111645 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.112789 4979 hybrid_clock.cc:648] HybridClock initialized: now 1790635491112766 us; error 39 us; skew 500 ppm
I20260928 22:44:51.114390 4979 webserver.cc:460] Webserver started at http://127.0.98.62:41197/ using document root <none> and password file <none>
I20260928 22:44:51.114620 4979 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:51.114667 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:51.114902 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:51.115974 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "dbd3ce31d9eb49cda0f38978714295a7"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "7fec20c2f3ca1d5eedaa4cbfe2d7a2ae"
server_key_iv: "3d3d644b4d85443f1ffd5f6d3197cbd0"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.116284 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "dbd3ce31d9eb49cda0f38978714295a7"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "7fec20c2f3ca1d5eedaa4cbfe2d7a2ae"
server_key_iv: "3d3d644b4d85443f1ffd5f6d3197cbd0"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.117580 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:51.118327 4993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.118525 4979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:51.118602 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "dbd3ce31d9eb49cda0f38978714295a7"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "7fec20c2f3ca1d5eedaa4cbfe2d7a2ae"
server_key_iv: "3d3d644b4d85443f1ffd5f6d3197cbd0"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.118678 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:51.122015 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:51.122277 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:51.122400 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:51.125891 4979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45935
I20260928 22:44:51.125926 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45935 every 8 connection(s)
I20260928 22:44:51.126259 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:51.126883 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:51.127799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 4979
I20260928 22:44:51.127882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:51.128129 392 external_mini_cluster.cc:1428] Setting key 55c60ae8d9e03774c7806695c8fd8884
I20260928 22:44:51.130240 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: Bootstrap starting.
I20260928 22:44:51.130748 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.128648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.98.62:45935)
0928 22:44:51.128928 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.128937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.128966 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:44:51.129476 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.129485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.129542 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.129697 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.130302 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.130306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.130307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.130311 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.130356 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.130369 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.130430 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.130498 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.130564 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260928 22:44:51.130942 5046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:51.131189 5046 log.cc:824] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:51.131781 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: No bootstrap required, opened a new log
I20260928 22:44:51.133107 5046 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } }
I20260928 22:44:51.133239 5046 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:51.133270 5046 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd3ce31d9eb49cda0f38978714295a7, State: Initialized, Role: FOLLOWER
I20260928 22:44:51.133389 5046 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } }
I20260928 22:44:51.133468 5046 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:51.133508 5046 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:51.133556 5046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:51.134032 5046 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } }
I20260928 22:44:51.134127 5046 leader_election.cc:302] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd3ce31d9eb49cda0f38978714295a7; no voters:
I20260928 22:44:51.134276 5046 leader_election.cc:288] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:51.134514 5046 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:51.134518 5051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:51.134652 5051 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [term 1 LEADER]: Becoming Leader. State: Replica: dbd3ce31d9eb49cda0f38978714295a7, State: Running, Role: LEADER
I20260928 22:44:51.134755 5051 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } }
I20260928 22:44:51.135046 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd3ce31d9eb49cda0f38978714295a7. Latest consensus state: current_term: 1 leader_uuid: "dbd3ce31d9eb49cda0f38978714295a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } } }
I20260928 22:44:51.135031 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd3ce31d9eb49cda0f38978714295a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd3ce31d9eb49cda0f38978714295a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45935 } } }
I20260928 22:44:51.135176 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:51.135183 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:51.135592 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:51.136018 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:51.137447 5060 catalog_manager.cc:1347] Generated new cluster ID: f1309734cfda472e837072d9987a8bed
I20260928 22:44:51.137498 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:51.157397 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:51.157593 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:51.161353 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: Generated new TSK 0
I20260928 22:44:51.161542 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:44:51.166699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--builtin_ntp_servers=127.0.98.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:44:51.247026 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.247257 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.247289 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.248472 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.248512 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.248546 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:51.249922 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.250172 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.250451 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.256016 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.256143 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.256022 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.256464 5070 server_base.cc:1034] running on GCE node
I20260928 22:44:51.256646 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.256883 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.258051 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790635491258031 us; error 33 us; skew 500 ppm
I20260928 22:44:51.259338 5070 webserver.cc:460] Webserver started at http://127.0.98.1:40391/ using document root <none> and password file <none>
I20260928 22:44:51.259577 5070 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:51.259625 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:51.259769 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:51.260926 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f31273a9527461b8a1b621939751cd9"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "4e9458d63f20dc3ef93e87ba4f19cb77"
server_key_iv: "90e53583bae79c53ca29a5f3609d016e"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.261277 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f31273a9527461b8a1b621939751cd9"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "4e9458d63f20dc3ef93e87ba4f19cb77"
server_key_iv: "90e53583bae79c53ca29a5f3609d016e"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.262480 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:51.263182 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.263353 5070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:51.263439 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "8f31273a9527461b8a1b621939751cd9"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "4e9458d63f20dc3ef93e87ba4f19cb77"
server_key_iv: "90e53583bae79c53ca29a5f3609d016e"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.263502 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:51.266812 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:51.267071 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:51.267189 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:51.267375 5070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:51.267649 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:51.267681 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.267717 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:51.267742 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.273155 5070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40311
I20260928 22:44:51.273201 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40311 every 8 connection(s)
I20260928 22:44:51.273499 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:51.276366 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.274204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37967 (local address 127.0.98.62:45935)
0928 22:44:51.274279 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.274281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.274489 (+ 208us) server_negotiation.cc:407] Connection header received
0928 22:44:51.274993 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.274995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.275000 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.275061 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.276145 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.276146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.276147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.276151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.276167 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.276171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.276217 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.276286 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.276299 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:44:51.276571 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.274154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45935 (local address 127.0.98.1:37967)
0928 22:44:51.274373 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.274407 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.274891 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.275141 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.275145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.275155 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.275998 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.275999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.276028 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.276059 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.276221 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.276226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.276229 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.276296 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.276358 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260928 22:44:51.277045 5198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45935
I20260928 22:44:51.277134 5198 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:51.277248 5198 heartbeater.cc:511] Master 127.0.98.62:45935 requested a full tablet report, sending...
I20260928 22:44:51.277734 5010 ts_manager.cc:194] Registered new tserver with Master: 8f31273a9527461b8a1b621939751cd9 (127.0.98.1:40311)
I20260928 22:44:51.282495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5070
I20260928 22:44:51.282577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:51.282697 392 external_mini_cluster.cc:1428] Setting key 64be72fc150af614d314ad906533e15d
I20260928 22:44:51.283488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--builtin_ntp_servers=127.0.98.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:44:51.362037 5201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.362258 5201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.362286 5201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.363442 5201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.363480 5201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.363513 5201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:51.364863 5201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.365097 5201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.365360 5201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.370158 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.370208 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.370246 5201 server_base.cc:1034] running on GCE node
W20260928 22:44:51.370158 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.370594 5201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.370819 5201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.371958 5201 hybrid_clock.cc:648] HybridClock initialized: now 1790635491371945 us; error 30 us; skew 500 ppm
I20260928 22:44:51.373221 5201 webserver.cc:460] Webserver started at http://127.0.98.2:37671/ using document root <none> and password file <none>
I20260928 22:44:51.373442 5201 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:51.373487 5201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:51.373602 5201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:51.374614 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "8bf64a7b8811475895a6e4664f63d0ce"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "8d3331e9c14064775bbacdb086ec50e4"
server_key_iv: "1db48852358a15c4337a4c47eb8fd5dc"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.374928 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8bf64a7b8811475895a6e4664f63d0ce"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "8d3331e9c14064775bbacdb086ec50e4"
server_key_iv: "1db48852358a15c4337a4c47eb8fd5dc"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.376093 5201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:44:51.376708 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.376879 5201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.376956 5201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "8bf64a7b8811475895a6e4664f63d0ce"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "8d3331e9c14064775bbacdb086ec50e4"
server_key_iv: "1db48852358a15c4337a4c47eb8fd5dc"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.377020 5201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:51.380226 5201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:51.380453 5201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:51.380565 5201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:51.380767 5201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:51.381052 5201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:51.381083 5201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.381108 5201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:51.381140 5201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.386265 5201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34637
I20260928 22:44:51.386323 5328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34637 every 8 connection(s)
I20260928 22:44:51.386595 5201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:51.387514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5201
I20260928 22:44:51.387585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:51.387748 392 external_mini_cluster.cc:1428] Setting key a7191bc3eb6a4e5d7190e79aacc67ace
I20260928 22:44:51.388545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--builtin_ntp_servers=127.0.98.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 22:44:51.389484 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.387240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38549 (local address 127.0.98.62:45935)
0928 22:44:51.387291 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.387293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.387508 (+ 215us) server_negotiation.cc:407] Connection header received
0928 22:44:51.388035 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.388037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.388042 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.388099 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.389262 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.389263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.389264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.389268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.389282 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.389286 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.389334 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.389402 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.389416 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:44:51.389678 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.387218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45935 (local address 127.0.98.2:38549)
0928 22:44:51.387400 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.387432 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.387940 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.388179 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.388183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.388192 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.389113 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.389114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.389149 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.389175 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.389336 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.389341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.389344 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.389413 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.389473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 22:44:51.390096 5329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45935
I20260928 22:44:51.390192 5329 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:51.390328 5329 heartbeater.cc:511] Master 127.0.98.62:45935 requested a full tablet report, sending...
I20260928 22:44:51.390658 5010 ts_manager.cc:194] Registered new tserver with Master: 8bf64a7b8811475895a6e4664f63d0ce (127.0.98.2:34637)
W20260928 22:44:51.466246 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.466455 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.466485 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.467645 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.467684 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.467717 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:51.469055 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.469283 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.469547 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.474309 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.474341 5332 server_base.cc:1034] running on GCE node
W20260928 22:44:51.474296 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.474314 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.474721 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.474951 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.476089 5332 hybrid_clock.cc:648] HybridClock initialized: now 1790635491476073 us; error 34 us; skew 500 ppm
I20260928 22:44:51.477331 5332 webserver.cc:460] Webserver started at http://127.0.98.3:45019/ using document root <none> and password file <none>
I20260928 22:44:51.477537 5332 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:51.477586 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:51.477718 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:51.478823 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca636a12a4804cd4b1589efdb586b616"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "0e294718952248a4c4f9d1ec38a6a5f5"
server_key_iv: "484a1514ff3d5f1361b4630d5c492640"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.479176 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca636a12a4804cd4b1589efdb586b616"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "0e294718952248a4c4f9d1ec38a6a5f5"
server_key_iv: "484a1514ff3d5f1361b4630d5c492640"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.480383 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:44:51.481045 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.481184 5332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.481258 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "ca636a12a4804cd4b1589efdb586b616"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "0e294718952248a4c4f9d1ec38a6a5f5"
server_key_iv: "484a1514ff3d5f1361b4630d5c492640"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.481324 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:51.484812 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:51.485054 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:51.485174 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:51.485356 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:51.485642 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:51.485677 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.485697 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:51.485708 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.490952 5332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38211
I20260928 22:44:51.491001 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38211 every 8 connection(s)
I20260928 22:44:51.491297 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:51.492905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5332
I20260928 22:44:51.493006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:51.493165 392 external_mini_cluster.cc:1428] Setting key 24036d32bf08628eeed3fbc6128c8fdf
I20260928 22:44:51.494189 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.491948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47707 (local address 127.0.98.62:45935)
0928 22:44:51.492013 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.492015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.492216 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:44:51.492670 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.492671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.492676 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.492729 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.493997 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.493997 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.493998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.494001 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.494016 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.494019 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.494064 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.494127 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.494138 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:44:51.494380 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.491931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45935 (local address 127.0.98.3:47707)
0928 22:44:51.492102 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.492146 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.492573 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.492820 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.492824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.492832 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.493812 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.493813 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.493843 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.493873 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.494071 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.494075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.494077 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.494132 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.494182 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 22:44:51.494803 5460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45935
I20260928 22:44:51.494876 5460 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:51.495016 5460 heartbeater.cc:511] Master 127.0.98.62:45935 requested a full tablet report, sending...
I20260928 22:44:51.495337 5010 ts_manager.cc:194] Registered new tserver with Master: ca636a12a4804cd4b1589efdb586b616 (127.0.98.3:38211)
I20260928 22:44:51.504787 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:44:51.507082 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.506079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.98.62:45935)
0928 22:44:51.506133 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.506135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.506203 (+ 68us) server_negotiation.cc:407] Connection header received
0928 22:44:51.506276 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.506276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.506282 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.506326 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.506847 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.506848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.506848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.506852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.506866 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.506870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.506916 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.506997 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.507011 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:51.508378 5010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:51.509924 5010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:51.513195 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.511795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38211 (local address 127.0.0.1:33854)
0928 22:44:51.512090 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.512101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.512148 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.512265 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.512271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.512277 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.512773 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.512774 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.512792 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.512815 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.513048 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.513054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.513057 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.513115 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.513127 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":99,"threads_started":1}
I20260928 22:44:51.513310 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.511780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40311 (local address 127.0.0.1:57048)
0928 22:44:51.511951 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.511991 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.512042 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.512482 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.512483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.512485 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.512919 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.512920 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.512927 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.512940 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.513203 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.513203 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.513204 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.513252 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.513263 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260928 22:44:51.513381 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.511900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.0.98.3:38211)
0928 22:44:51.512127 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.512132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.512141 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:51.512194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.512197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.512205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.512267 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.512873 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.512880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.512880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.512884 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.512914 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.512940 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.513025 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.513290 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.513306 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260928 22:44:51.513819 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.511954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.0.98.1:40311)
0928 22:44:51.512358 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.512363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.512372 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:51.512396 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.512398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.512406 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.512482 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.513017 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.513026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.513026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.513031 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.513064 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.513091 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.513192 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.513721 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.513738 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":68,"threads_started":1}
I20260928 22:44:51.514030 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.511981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34637 (local address 127.0.0.1:55980)
0928 22:44:51.512203 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.512212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.512250 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.513248 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.513249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.513251 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.513700 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.513700 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.513706 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.513717 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.513920 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.513921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.513922 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.513960 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.513972 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260928 22:44:51.514055 5394 tablet_service.cc:1467] Processing CreateTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d (DEFAULT_TABLE table=test-table [id=d73b59662bea4832a5e6378822df02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:51.514215 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.512184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.98.2:34637)
0928 22:44:51.512358 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.512363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.512372 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:51.512396 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.512398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.512406 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.512990 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.513788 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.513795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.513795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.513799 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.513828 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.513848 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.513894 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.514122 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.514137 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260928 22:44:51.514417 5394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd42d2730e1b433d80f90a03c5a0da1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:51.514380 5132 tablet_service.cc:1467] Processing CreateTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d (DEFAULT_TABLE table=test-table [id=d73b59662bea4832a5e6378822df02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:51.514689 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd42d2730e1b433d80f90a03c5a0da1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:51.517601 5474 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Bootstrap starting.
I20260928 22:44:51.518131 5263 tablet_service.cc:1467] Processing CreateTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d (DEFAULT_TABLE table=test-table [id=d73b59662bea4832a5e6378822df02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:51.518236 5474 tablet_bootstrap.cc:654] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:51.518394 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd42d2730e1b433d80f90a03c5a0da1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:51.518501 5474 log.cc:824] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:51.518775 5475 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Bootstrap starting.
I20260928 22:44:51.519156 5474 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: No bootstrap required, opened a new log
I20260928 22:44:51.519227 5474 ts_tablet_manager.cc:1397] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:51.519273 5475 tablet_bootstrap.cc:654] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:51.519520 5475 log.cc:824] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:51.520210 5475 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: No bootstrap required, opened a new log
I20260928 22:44:51.520264 5475 ts_tablet_manager.cc:1397] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:51.521588 5478 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Bootstrap starting.
I20260928 22:44:51.521651 5474 raft_consensus.cc:348] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.521812 5474 raft_consensus.cc:374] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:51.521854 5474 raft_consensus.cc:729] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f31273a9527461b8a1b621939751cd9, State: Initialized, Role: FOLLOWER
I20260928 22:44:51.521968 5474 consensus_queue.cc:260] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.522197 5478 tablet_bootstrap.cc:654] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:51.522218 5474 ts_tablet_manager.cc:1428] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:44:51.522357 5198 heartbeater.cc:503] Master 127.0.98.62:45935 was elected leader, sending a full tablet report...
I20260928 22:44:51.522305 5475 raft_consensus.cc:348] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.522446 5475 raft_consensus.cc:374] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:51.522475 5478 log.cc:824] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:51.522487 5475 raft_consensus.cc:729] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca636a12a4804cd4b1589efdb586b616, State: Initialized, Role: FOLLOWER
I20260928 22:44:51.522591 5475 consensus_queue.cc:260] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.522823 5475 ts_tablet_manager.cc:1428] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:44:51.522933 5460 heartbeater.cc:503] Master 127.0.98.62:45935 was elected leader, sending a full tablet report...
I20260928 22:44:51.523144 5478 tablet_bootstrap.cc:492] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: No bootstrap required, opened a new log
I20260928 22:44:51.523206 5478 ts_tablet_manager.cc:1397] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20260928 22:44:51.523958 5199 tablet.cc:2369] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:51.525183 5478 raft_consensus.cc:348] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.525302 5478 raft_consensus.cc:374] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:51.525331 5478 raft_consensus.cc:729] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf64a7b8811475895a6e4664f63d0ce, State: Initialized, Role: FOLLOWER
I20260928 22:44:51.525437 5478 consensus_queue.cc:260] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.525640 5478 ts_tablet_manager.cc:1428] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:44:51.525694 5329 heartbeater.cc:503] Master 127.0.98.62:45935 was elected leader, sending a full tablet report...
I20260928 22:44:51.616050 5482 raft_consensus.cc:482] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:51.616191 5482 raft_consensus.cc:504] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.616552 5482 leader_election.cc:288] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f31273a9527461b8a1b621939751cd9 (127.0.98.1:40311), ca636a12a4804cd4b1589efdb586b616 (127.0.98.3:38211)
I20260928 22:44:51.617931 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.616857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41527 (local address 127.0.98.1:40311)
0928 22:44:51.616896 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.616898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.617058 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:44:51.617086 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.617087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.617093 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.617144 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.617705 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.617706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.617707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.617711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.617731 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.617736 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.617792 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.617849 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.617861 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:51.617933 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.616821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35575 (local address 127.0.98.3:38211)
0928 22:44:51.616878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.616881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.616895 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:44:51.617018 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.617019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.617026 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.617080 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.617724 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.617724 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.617725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.617728 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.617744 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.617748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.617799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.617847 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.617860 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:44:51.617944 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.616777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38211 (local address 127.0.98.2:35575)
0928 22:44:51.616849 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.616859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.616903 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.617107 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.617107 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.617110 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.617641 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.617642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.617646 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.617654 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.617802 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.617803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.617803 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.617843 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.617860 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
I20260928 22:44:51.617954 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.616780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40311 (local address 127.0.98.2:41527)
0928 22:44:51.616963 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.616973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.617031 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.617146 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.617146 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.617148 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:51.617593 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:44:51.617594 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:51.617602 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:44:51.617618 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:51.617802 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:51.617803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:51.617803 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:44:51.617842 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 22:44:51.617858 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":22,"thread_start_us":108,"threads_started":1}
I20260928 22:44:51.618335 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd42d2730e1b433d80f90a03c5a0da1d" candidate_uuid: "8bf64a7b8811475895a6e4664f63d0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca636a12a4804cd4b1589efdb586b616" is_pre_election: true
I20260928 22:44:51.618444 5152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd42d2730e1b433d80f90a03c5a0da1d" candidate_uuid: "8bf64a7b8811475895a6e4664f63d0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f31273a9527461b8a1b621939751cd9" is_pre_election: true
I20260928 22:44:51.618496 5414 raft_consensus.cc:2454] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf64a7b8811475895a6e4664f63d0ce in term 0.
I20260928 22:44:51.618572 5152 raft_consensus.cc:2454] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf64a7b8811475895a6e4664f63d0ce in term 0.
I20260928 22:44:51.618791 5217 leader_election.cc:302] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf64a7b8811475895a6e4664f63d0ce, 8f31273a9527461b8a1b621939751cd9; no voters:
I20260928 22:44:51.618968 5482 raft_consensus.cc:2781] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:51.619053 5482 raft_consensus.cc:482] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:51.619081 5482 raft_consensus.cc:3037] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:51.619758 5482 raft_consensus.cc:504] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.619882 5482 leader_election.cc:288] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [CANDIDATE]: Term 1 election: Requested vote from peers 8f31273a9527461b8a1b621939751cd9 (127.0.98.1:40311), ca636a12a4804cd4b1589efdb586b616 (127.0.98.3:38211)
I20260928 22:44:51.620029 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd42d2730e1b433d80f90a03c5a0da1d" candidate_uuid: "8bf64a7b8811475895a6e4664f63d0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca636a12a4804cd4b1589efdb586b616"
I20260928 22:44:51.620072 5152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd42d2730e1b433d80f90a03c5a0da1d" candidate_uuid: "8bf64a7b8811475895a6e4664f63d0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f31273a9527461b8a1b621939751cd9"
I20260928 22:44:51.620107 5414 raft_consensus.cc:3037] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:51.620137 5152 raft_consensus.cc:3037] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:51.620834 5152 raft_consensus.cc:2454] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf64a7b8811475895a6e4664f63d0ce in term 1.
I20260928 22:44:51.620857 5414 raft_consensus.cc:2454] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf64a7b8811475895a6e4664f63d0ce in term 1.
I20260928 22:44:51.621019 5217 leader_election.cc:302] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8bf64a7b8811475895a6e4664f63d0ce, 8f31273a9527461b8a1b621939751cd9, ca636a12a4804cd4b1589efdb586b616; no voters:
I20260928 22:44:51.621126 5482 raft_consensus.cc:2781] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:51.621297 5482 raft_consensus.cc:686] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 1 LEADER]: Becoming Leader. State: Replica: 8bf64a7b8811475895a6e4664f63d0ce, State: Running, Role: LEADER
I20260928 22:44:51.621383 5482 consensus_queue.cc:237] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260928 22:44:51.622225 5010 catalog_manager.cc:5495] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf64a7b8811475895a6e4664f63d0ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8bf64a7b8811475895a6e4664f63d0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf64a7b8811475895a6e4664f63d0ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } health_report { overall_health: HEALTHY } } }
I20260928 22:44:51.628523 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.627745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.0.98.2:34637)
0928 22:44:51.627785 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.627787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.627796 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:51.627866 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.627866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.627878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.627911 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.628387 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.628388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.628388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.628390 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.628400 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.628403 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.628433 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.628467 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.628476 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:51.630110 5414 raft_consensus.cc:1261] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 1 FOLLOWER]: Refusing update from remote peer 8bf64a7b8811475895a6e4664f63d0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:51.630110 5152 raft_consensus.cc:1261] T fd42d2730e1b433d80f90a03c5a0da1d P 8f31273a9527461b8a1b621939751cd9 [term 1 FOLLOWER]: Refusing update from remote peer 8bf64a7b8811475895a6e4664f63d0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:51.630420 5484 consensus_queue.cc:1035] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f31273a9527461b8a1b621939751cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:51.630532 5482 consensus_queue.cc:1035] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca636a12a4804cd4b1589efdb586b616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:51.634948 5009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:51.635040 5009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:51.635726 5009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbd3ce31d9eb49cda0f38978714295a7: Sending DeleteTablet for 3 replicas of tablet fd42d2730e1b433d80f90a03c5a0da1d
I20260928 22:44:51.636035 5132 tablet_service.cc:1514] Processing DeleteTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:51 UTC) from {username='slave'} at 127.0.0.1:57048
I20260928 22:44:51.636044 5394 tablet_service.cc:1514] Processing DeleteTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:51 UTC) from {username='slave'} at 127.0.0.1:33854
I20260928 22:44:51.636322 5497 tablet_replica.cc:330] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: stopping tablet replica
I20260928 22:44:51.636332 5263 tablet_service.cc:1514] Processing DeleteTablet for tablet fd42d2730e1b433d80f90a03c5a0da1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:51 UTC) from {username='slave'} at 127.0.0.1:55980
I20260928 22:44:51.636381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5070
I20260928 22:44:51.637566 5498 tablet_replica.cc:330] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: stopping tablet replica
I20260928 22:44:51.637655 5498 raft_consensus.cc:2229] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:51.637729 5498 raft_consensus.cc:2258] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:51.638113 5498 ts_tablet_manager.cc:1905] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:51.639597 5498 ts_tablet_manager.cc:1918] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:51.639663 5498 log.cc:1196] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/fd42d2730e1b433d80f90a03c5a0da1d
I20260928 22:44:51.639880 5498 ts_tablet_manager.cc:1939] T fd42d2730e1b433d80f90a03c5a0da1d P 8bf64a7b8811475895a6e4664f63d0ce: Deleting consensus metadata
I20260928 22:44:51.640208 4995 catalog_manager.cc:4841] TS 8bf64a7b8811475895a6e4664f63d0ce (127.0.98.2:34637): tablet fd42d2730e1b433d80f90a03c5a0da1d (table test-table [id=d73b59662bea4832a5e6378822df02bb]) successfully deleted
W20260928 22:44:51.641817 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40311 (error 108)
I20260928 22:44:51.642210 5497 raft_consensus.cc:2229] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:51.642246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5201
I20260928 22:44:51.642333 5497 raft_consensus.cc:2258] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:51.642681 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.642581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40311 (local address 127.0.0.1:57062)
0928 22:44:51.642623 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.642653 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 22:44:51.642762 4995 catalog_manager.cc:4584] TS 8f31273a9527461b8a1b621939751cd9 (127.0.98.1:40311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd42d2730e1b433d80f90a03c5a0da1d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40311: connect: Connection refused (error 111)
I20260928 22:44:51.643323 5497 ts_tablet_manager.cc:1905] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:51.644436 5497 ts_tablet_manager.cc:1918] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:51.644495 5497 log.cc:1196] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/fd42d2730e1b433d80f90a03c5a0da1d
I20260928 22:44:51.644691 5497 ts_tablet_manager.cc:1939] T fd42d2730e1b433d80f90a03c5a0da1d P ca636a12a4804cd4b1589efdb586b616: Deleting consensus metadata
I20260928 22:44:51.645083 4994 catalog_manager.cc:4841] TS ca636a12a4804cd4b1589efdb586b616 (127.0.98.3:38211): tablet fd42d2730e1b433d80f90a03c5a0da1d (table test-table [id=d73b59662bea4832a5e6378822df02bb]) successfully deleted
I20260928 22:44:51.647306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5332
I20260928 22:44:51.651141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 4979
2026-09-28T22:44:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (651 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 22:44:51.669054 392 mini_oidc.cc:152] Starting JWKS server
I20260928 22:44:51.670051 392 webserver.cc:460] Webserver started at https://127.0.0.1:33405/ using document root <none> and password file <none>
I20260928 22:44:51.670127 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:44:51.670252 392 webserver.cc:460] Webserver started at http://127.0.0.1:36855/ using document root <none> and password file <none>
2026-09-28T22:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:51Z Disabled control of system clock
I20260928 22:44:51.677193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41923
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41923
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/testchainca.cert with env {}
W20260928 22:44:51.755163 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.755373 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.755393 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.756522 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:51.756556 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.756567 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.756578 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:51.756588 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:44:51.756599 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:44:51.756609 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33405/jwks.json
I20260928 22:44:51.758185 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.758387 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.758558 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.763027 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.763032 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.763115 5505 server_base.cc:1034] running on GCE node
W20260928 22:44:51.763027 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.763449 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.763702 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.764840 5505 hybrid_clock.cc:648] HybridClock initialized: now 1790635491764822 us; error 37 us; skew 500 ppm
I20260928 22:44:51.766074 5505 webserver.cc:460] Webserver started at http://127.0.98.62:45609/ using document root <none> and password file <none>
I20260928 22:44:51.766305 5505 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:51.766352 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:51.766471 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:51.767524 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "9190768dd14046b98977ee3f662306dd"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "d64343db9ff6b48c77e3ee2c295dac54"
server_key_iv: "a952fb4696c5f7080227d1c2ad121ec3"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.767848 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "9190768dd14046b98977ee3f662306dd"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "d64343db9ff6b48c77e3ee2c295dac54"
server_key_iv: "a952fb4696c5f7080227d1c2ad121ec3"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.769084 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:51.769793 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:51.769958 5505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:51.770057 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "9190768dd14046b98977ee3f662306dd"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "d64343db9ff6b48c77e3ee2c295dac54"
server_key_iv: "a952fb4696c5f7080227d1c2ad121ec3"
server_key_version: "encryptionkey@0"
I20260928 22:44:51.770123 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:51.791042 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:51.791355 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:51.791489 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:51.795228 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41923
I20260928 22:44:51.795262 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41923 every 8 connection(s)
I20260928 22:44:51.795593 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:51.796519 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:51.798833 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd: Bootstrap starting.
I20260928 22:44:51.799405 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:51.799676 5572 log.cc:824] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:51.800308 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd: No bootstrap required, opened a new log
I20260928 22:44:51.801190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5505
I20260928 22:44:51.801286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:51.801447 392 external_mini_cluster.cc:1428] Setting key fc6969f1b5dc9ea65dc9c4060377867e
I20260928 22:44:51.802075 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } }
I20260928 22:44:51.802227 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:51.802258 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9190768dd14046b98977ee3f662306dd, State: Initialized, Role: FOLLOWER
I20260928 22:44:51.802384 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } }
I20260928 22:44:51.802461 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:51.802493 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:51.802538 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:51.803050 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } }
I20260928 22:44:51.803144 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9190768dd14046b98977ee3f662306dd; no voters:
I20260928 22:44:51.803300 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:51.803341 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:51.803445 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [term 1 LEADER]: Becoming Leader. State: Replica: 9190768dd14046b98977ee3f662306dd, State: Running, Role: LEADER
I20260928 22:44:51.803524 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } }
I20260928 22:44:51.803561 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:51.803840 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9190768dd14046b98977ee3f662306dd. Latest consensus state: current_term: 1 leader_uuid: "9190768dd14046b98977ee3f662306dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } } }
I20260928 22:44:51.803908 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:51.804040 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9190768dd14046b98977ee3f662306dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9190768dd14046b98977ee3f662306dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41923 } } }
I20260928 22:44:51.804098 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:51.804210 5582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:51.804735 5582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:51.806703 5582 catalog_manager.cc:1347] Generated new cluster ID: ea04024d14c346b789136815d9275852
I20260928 22:44:51.806759 5582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:51.807209 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.801938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.0.98.62:41923)
0928 22:44:51.802122 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.802129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.802153 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:44:51.802246 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.802251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.802317 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.802435 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:51.802662 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.804133 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.805144 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.805625 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.805737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:51.806447 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:51.806450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:51.806451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:51.806655 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:51.806695 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:51.806709 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:51.806775 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:51.806934 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 22:44:51.807017 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260928 22:44:51.808073 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:44:51.822557 5582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:51.823303 5582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:51.836903 5582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9190768dd14046b98977ee3f662306dd: Generated new TSK 0
I20260928 22:44:51.837127 5582 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:44:51.865777 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 22:44:51.865886 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 22:44:51.866101 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:51.851236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.62:41923)
0928 22:44:51.851280 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:51.851282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:51.851364 (+ 82us) server_negotiation.cc:407] Connection header received
0928 22:44:51.851477 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:51.851478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:51.851487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:51.851533 (+ 46us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:44:51.851709 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.852189 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.853320 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.853507 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.853552 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:51.853769 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:44:51.865869 (+ 12100us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:44:51.866048 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:44:51.866142 5576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 22:44:51.866588 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:51.851142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41923 (local address 127.0.0.1:59992)
0928 22:44:51.851294 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:51.851305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:51.851378 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:51.851540 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:51.851541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:51.851543 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:44:51.851624 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:51.851626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.852274 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.852274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:51.853235 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:51.853238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:51.853568 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:51.853571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:51.853636 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:51.853661 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:44:51.866249 (+ 12588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:44:51.866352 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 22:44:51.867009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5505
2026-09-28T22:44:51Z chronyd exiting
I20260928 22:44:51.902441 392 test_util.cc:175] -----------------------------------------------
I20260928 22:44:51.902518 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635468897675-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 22:44:51.903239 392 mini_oidc.cc:152] Starting JWKS server
I20260928 22:44:51.903892 392 webserver.cc:460] Webserver started at https://127.0.0.1:45147/ using document root <none> and password file <none>
I20260928 22:44:51.903934 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:44:51.904064 392 webserver.cc:460] Webserver started at http://127.0.0.1:46565/ using document root <none> and password file <none>
2026-09-28T22:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:51Z Disabled control of system clock
I20260928 22:44:51.910727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33945
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45147/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/testchainca.cert with env {}
W20260928 22:44:51.989248 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:51.989456 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:51.989485 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:51.990641 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:51.990682 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:51.990700 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:51.990715 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:51.990731 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:44:51.990747 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:44:51.990769 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45147/jwks.json
I20260928 22:44:51.992266 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45147/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:51.992507 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:51.992707 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:51.997180 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.997157 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:51.997196 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:51.997376 5608 server_base.cc:1034] running on GCE node
I20260928 22:44:51.997581 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:51.997828 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:51.998970 5608 hybrid_clock.cc:648] HybridClock initialized: now 1790635491998959 us; error 30 us; skew 500 ppm
I20260928 22:44:52.003502 5608 webserver.cc:460] Webserver started at http://127.0.98.62:40227/ using document root <none> and password file <none>
I20260928 22:44:52.003719 5608 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:52.003764 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:52.003885 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:52.004998 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "2814114b6de64518960fbb448356a096"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "2efef8bf4610ec4135cde2c780c495d7"
server_key_iv: "55f29607501b6385b0a8d81e550ff6a2"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.005321 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "2814114b6de64518960fbb448356a096"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "2efef8bf4610ec4135cde2c780c495d7"
server_key_iv: "55f29607501b6385b0a8d81e550ff6a2"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.006497 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:52.007162 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:52.007308 5608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:52.007387 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "2814114b6de64518960fbb448356a096"
format_stamp: "Formatted at 2026-09-28 22:44:51 on dist-test-slave-pxr1"
server_key: "2efef8bf4610ec4135cde2c780c495d7"
server_key_iv: "55f29607501b6385b0a8d81e550ff6a2"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.007447 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:52.017428 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:52.017712 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:52.017831 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:52.021425 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33945
I20260928 22:44:52.021463 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33945 every 8 connection(s)
I20260928 22:44:52.021786 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:52.022670 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:52.024449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5608
I20260928 22:44:52.024529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:52.024691 392 external_mini_cluster.cc:1428] Setting key 04d4d2956c3ac66b1fe7c8edaaeebffd
I20260928 22:44:52.024959 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: Bootstrap starting.
I20260928 22:44:52.025574 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:52.025820 5675 log.cc:824] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:52.026466 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: No bootstrap required, opened a new log
I20260928 22:44:52.028436 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:44:52.028600 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:52.028637 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2814114b6de64518960fbb448356a096, State: Initialized, Role: FOLLOWER
I20260928 22:44:52.028802 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:44:52.028878 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:52.028918 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:52.028955 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:52.029449 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:44:52.029547 5675 leader_election.cc:302] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2814114b6de64518960fbb448356a096; no voters:
I20260928 22:44:52.029693 5675 leader_election.cc:288] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:52.029718 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:52.029856 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [term 1 LEADER]: Becoming Leader. State: Replica: 2814114b6de64518960fbb448356a096, State: Running, Role: LEADER
I20260928 22:44:52.029927 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:44:52.029930 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:52.030190 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.025183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.62:33945)
0928 22:44:52.025362 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.025370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.025394 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:44:52.025847 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.025854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.025939 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.026101 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:52.026256 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.027167 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.028252 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.028647 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.028779 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.029431 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:52.029436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:52.029436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:52.029617 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:52.029658 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:52.029672 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:52.029726 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:52.029908 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 22:44:52.029994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 22:44:52.031204 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2814114b6de64518960fbb448356a096. Latest consensus state: current_term: 1 leader_uuid: "2814114b6de64518960fbb448356a096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } } }
I20260928 22:44:52.031299 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.031456 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 22:44:52.031736 5695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:44:52.031791 5695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:44:52.031324 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2814114b6de64518960fbb448356a096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2814114b6de64518960fbb448356a096" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } } }
I20260928 22:44:52.032060 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.032174 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:52.032344 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:52.033758 5697 catalog_manager.cc:1347] Generated new cluster ID: 4f3c9ea6bccd40ef8d5c3c54295140ee
I20260928 22:44:52.033811 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:52.049780 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:52.050274 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:52.058566 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2814114b6de64518960fbb448356a096: Generated new TSK 0
I20260928 22:44:52.058748 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:44:52.081653 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 22:44:52.081722 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 22:44:52.082106 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.067905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:33945)
0928 22:44:52.067949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.067951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.068043 (+ 92us) server_negotiation.cc:407] Connection header received
0928 22:44:52.068090 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.068091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.068102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.068181 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:44:52.068274 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.069089 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.069805 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.070129 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.070193 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.070320 (+ 127us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:44:52.081868 (+ 11548us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45147/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:44:52.082013 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45147/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 22:44:52.082165 5678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45147/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 22:44:52.082165 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:52.067848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33945 (local address 127.0.0.1:56628)
0928 22:44:52.067969 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:52.067986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:52.068039 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:52.068176 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:52.068177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:52.068178 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:44:52.068224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.068225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.069155 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.069156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.069733 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.069735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.070188 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.070192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.070229 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.070234 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:44:52.081974 (+ 11740us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45147/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:44:52.082087 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45147/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260928 22:44:52.082434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5608
2026-09-28T22:44:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 22:44:52.118909 392 mini_oidc.cc:152] Starting JWKS server
I20260928 22:44:52.119553 392 webserver.cc:460] Webserver started at https://127.0.0.1:33419/ using document root <none> and password file <none>
I20260928 22:44:52.119599 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:44:52.119738 392 webserver.cc:460] Webserver started at http://127.0.0.1:40081/ using document root <none> and password file <none>
2026-09-28T22:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:52Z Disabled control of system clock
I20260928 22:44:52.126506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37567
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:44:52.204030 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:52.204246 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:52.204275 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:52.205482 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:52.205526 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:52.205542 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:52.205557 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:52.205572 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:44:52.205588 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:44:52.205607 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33419/jwks.json
I20260928 22:44:52.206985 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:52.207237 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:52.207506 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:44:52.212201 5711 server_base.cc:1034] running on GCE node
W20260928 22:44:52.212176 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:52.212185 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:52.212314 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:52.212654 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:52.213007 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:52.214156 5711 hybrid_clock.cc:648] HybridClock initialized: now 1790635492214143 us; error 28 us; skew 500 ppm
I20260928 22:44:52.215516 5711 webserver.cc:460] Webserver started at http://127.0.98.62:44199/ using document root <none> and password file <none>
I20260928 22:44:52.215739 5711 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:52.215791 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:52.215920 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:52.217025 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "b8acaa5c4c1f4501abf093cf82c1d088"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "2eb4ab94a881a48b6d2cbc79a03089b8"
server_key_iv: "fc90a3386269f7da57bbd4b25eabbd7b"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.217356 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8acaa5c4c1f4501abf093cf82c1d088"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "2eb4ab94a881a48b6d2cbc79a03089b8"
server_key_iv: "fc90a3386269f7da57bbd4b25eabbd7b"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.218621 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:52.219283 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:52.219425 5711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:52.219507 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "b8acaa5c4c1f4501abf093cf82c1d088"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "2eb4ab94a881a48b6d2cbc79a03089b8"
server_key_iv: "fc90a3386269f7da57bbd4b25eabbd7b"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.219561 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:52.240635 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:52.240976 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:52.241122 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:52.244679 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37567
I20260928 22:44:52.244741 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37567 every 8 connection(s)
I20260928 22:44:52.245074 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:52.245709 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:52.248251 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088: Bootstrap starting.
I20260928 22:44:52.248808 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:52.249055 5778 log.cc:824] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:52.249627 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088: No bootstrap required, opened a new log
I20260928 22:44:52.250530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5711
I20260928 22:44:52.250635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:52.250798 392 external_mini_cluster.cc:1428] Setting key 049e81be82ab8ea1470696538a1aa392
I20260928 22:44:52.251585 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } }
I20260928 22:44:52.251726 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:52.251757 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8acaa5c4c1f4501abf093cf82c1d088, State: Initialized, Role: FOLLOWER
I20260928 22:44:52.251901 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } }
I20260928 22:44:52.252004 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:52.252039 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:52.252080 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:52.252600 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } }
I20260928 22:44:52.252687 5778 leader_election.cc:302] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8acaa5c4c1f4501abf093cf82c1d088; no voters:
I20260928 22:44:52.252862 5778 leader_election.cc:288] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:52.252903 5783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:52.253003 5783 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [term 1 LEADER]: Becoming Leader. State: Replica: b8acaa5c4c1f4501abf093cf82c1d088, State: Running, Role: LEADER
I20260928 22:44:52.253091 5783 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } }
I20260928 22:44:52.253122 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:52.253396 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8acaa5c4c1f4501abf093cf82c1d088. Latest consensus state: current_term: 1 leader_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } } }
I20260928 22:44:52.253482 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.253410 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8acaa5c4c1f4501abf093cf82c1d088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37567 } } }
I20260928 22:44:52.253631 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.253669 5788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:52.254179 5788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:52.256328 5788 catalog_manager.cc:1347] Generated new cluster ID: ce987604e2d944ba93940723ef84988e
I20260928 22:44:52.256381 5788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:52.257098 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.251323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.98.62:37567)
0928 22:44:52.251518 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.251526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.251551 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:44:52.251647 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.251652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.251704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.251826 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:52.252096 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.253645 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.254777 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.255359 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.255472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.256217 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:52.256221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:52.256221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:52.256427 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:52.256469 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:52.256484 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:52.256771 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:52.256805 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 22:44:52.256887 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260928 22:44:52.258001 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:44:52.262601 5788 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:52.263306 5788 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:52.272109 5788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8acaa5c4c1f4501abf093cf82c1d088: Generated new TSK 0
I20260928 22:44:52.272333 5788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:44:52.284353 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.277434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.0.98.62:37567)
0928 22:44:52.277478 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.277481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.279559 (+ 2078us) server_negotiation.cc:407] Connection header received
0928 22:44:52.279731 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.279733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.279751 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.279835 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:44:52.280005 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.280825 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.282425 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.282724 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.282788 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.283097 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:44:52.284131 (+ 1034us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:44:52.284268 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:44:52.284413 5782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 22:44:52.286307 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:52.277217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37567 (local address 127.0.0.1:56982)
0928 22:44:52.279485 (+ 2268us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:52.279497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:52.279605 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:52.279855 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:52.279856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:52.279858 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:44:52.279947 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.279949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.280925 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.280926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.282315 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.282319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.282835 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.282844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.282919 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.282927 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:44:52.284237 (+ 1310us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:44:52.286214 (+ 1977us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":2236,"mutex_wait_us":1801,"thread_start_us":2102,"threads_started":1}
I20260928 22:44:52.286473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5711
E20260928 22:44:52.296994 5808 webserver.cc:533] Webserver: sslize failed:
E20260928 22:44:52.297070 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T22:44:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T22:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:52Z Disabled control of system clock
I20260928 22:44:52.331593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33965
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/testchainca.cert with env {}
W20260928 22:44:52.410104 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:52.410315 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:52.410337 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:52.411542 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:52.411589 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:52.411602 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:52.411612 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:52.411624 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:44:52.411648 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:44:52.411664 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 22:44:52.413223 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:52.413466 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:52.413663 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:52.418293 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:52.418290 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:52.418298 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:52.418491 5812 server_base.cc:1034] running on GCE node
I20260928 22:44:52.418740 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:52.419044 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:52.420203 5812 hybrid_clock.cc:648] HybridClock initialized: now 1790635492420189 us; error 30 us; skew 500 ppm
I20260928 22:44:52.421458 5812 webserver.cc:460] Webserver started at http://127.0.98.62:40157/ using document root <none> and password file <none>
I20260928 22:44:52.421675 5812 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:52.421723 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:52.421847 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:52.422933 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "c32f379fed5c447b92f75fb8b8cb6724"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "bf606b7ac32cc6b0336eacdb92a45b17"
server_key_iv: "1735af21e2e904e7defe1cd95da0c3c0"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.423275 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "c32f379fed5c447b92f75fb8b8cb6724"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "bf606b7ac32cc6b0336eacdb92a45b17"
server_key_iv: "1735af21e2e904e7defe1cd95da0c3c0"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.424485 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:52.425271 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:52.425442 5812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:52.425529 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "c32f379fed5c447b92f75fb8b8cb6724"
format_stamp: "Formatted at 2026-09-28 22:44:52 on dist-test-slave-pxr1"
server_key: "bf606b7ac32cc6b0336eacdb92a45b17"
server_key_iv: "1735af21e2e904e7defe1cd95da0c3c0"
server_key_version: "encryptionkey@0"
I20260928 22:44:52.425596 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:52.434911 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:52.435212 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:52.435340 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:52.439170 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33965
I20260928 22:44:52.439196 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33965 every 8 connection(s)
I20260928 22:44:52.439507 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:52.440393 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:52.442610 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724: Bootstrap starting.
I20260928 22:44:52.443118 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:52.443336 5879 log.cc:824] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:52.443931 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724: No bootstrap required, opened a new log
I20260928 22:44:52.445346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5812
I20260928 22:44:52.445451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:52.445638 392 external_mini_cluster.cc:1428] Setting key 954a4150e906ec9a194486f1b88e713d
I20260928 22:44:52.445679 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260928 22:44:52.445839 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:52.445864 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32f379fed5c447b92f75fb8b8cb6724, State: Initialized, Role: FOLLOWER
I20260928 22:44:52.446014 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260928 22:44:52.446094 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:52.446122 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:52.446172 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:52.446755 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260928 22:44:52.446864 5879 leader_election.cc:302] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32f379fed5c447b92f75fb8b8cb6724; no voters:
I20260928 22:44:52.447012 5879 leader_election.cc:288] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:52.447166 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:52.447248 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:52.447281 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [term 1 LEADER]: Becoming Leader. State: Replica: c32f379fed5c447b92f75fb8b8cb6724, State: Running, Role: LEADER
I20260928 22:44:52.447368 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } }
I20260928 22:44:52.447687 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32f379fed5c447b92f75fb8b8cb6724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260928 22:44:52.447757 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.447893 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32f379fed5c447b92f75fb8b8cb6724. Latest consensus state: current_term: 1 leader_uuid: "c32f379fed5c447b92f75fb8b8cb6724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f379fed5c447b92f75fb8b8cb6724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33965 } } }
I20260928 22:44:52.447939 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:52.448601 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:52.449216 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:52.450481 5892 catalog_manager.cc:1347] Generated new cluster ID: e0e647e904dc40bc991c0836381ee225
I20260928 22:44:52.450533 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:52.451313 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.446110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.0.98.62:33965)
0928 22:44:52.446300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.446308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.446380 (+ 72us) server_negotiation.cc:407] Connection header received
0928 22:44:52.446527 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.446532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.446592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.446698 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:52.446956 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.448200 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.449255 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.449808 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.449919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.450619 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:52.450622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:52.450623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:44:52.450818 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:52.450856 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:44:52.450868 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:52.450925 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:52.451030 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 22:44:52.451106 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260928 22:44:52.453444 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:44:52.455035 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:52.455484 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:52.462186 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c32f379fed5c447b92f75fb8b8cb6724: Generated new TSK 0
I20260928 22:44:52.462368 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:44:52.530663 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:52.471740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.0.98.62:33965)
0928 22:44:52.471785 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:52.471787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:52.471855 (+ 68us) server_negotiation.cc:407] Connection header received
0928 22:44:52.471907 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:52.471908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:52.471919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:52.472006 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:44:52.472167 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.473003 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.473788 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.474097 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.474166 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.474310 (+ 144us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:44:52.530372 (+ 56062us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:44:52.530580 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 22:44:52.530736 5883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 22:44:52.530794 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:52.471659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33965 (local address 127.0.0.1:50766)
0928 22:44:52.471784 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:52.471793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:52.471852 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:52.472031 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:52.472033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:52.472035 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:44:52.472094 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.472096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.473099 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.473100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.473708 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:52.473710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:52.474167 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:52.474172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:52.474212 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:52.474217 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:44:52.530539 (+ 56322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:44:52.530690 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260928 22:44:52.531299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5812
2026-09-28T22:44:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (225 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:52 dist-test-slave-pxr1 krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:52 dist-test-slave-pxr1 krb5kdc[5920](info): set up 2 sockets
Sep 28 22:44:52 dist-test-slave-pxr1 krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260928 22:44:53.652752 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:53 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:44:53.761554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:53.840263 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:53.840488 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:53.840519 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:53.841733 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:53.841775 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:53.841794 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:53.841809 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:53.841823 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:53.843345 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:53.843595 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:53.843798 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:53.848372 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:53.848378 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:53.848436 5933 server_base.cc:1034] running on GCE node
W20260928 22:44:53.848381 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:53.848834 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:53.849131 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:53.850270 5933 hybrid_clock.cc:648] HybridClock initialized: now 1790635493850256 us; error 29 us; skew 500 ppm
Sep 28 22:44:53 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:53.852612 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:44:53.853103 5933 webserver.cc:460] Webserver started at http://127.0.98.62:43453/ using document root <none> and password file <none>
I20260928 22:44:53.853317 5933 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:53.853359 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:53.853483 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:53.854535 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "77a48f0a3d1043019b28f5921d5cfd98"
format_stamp: "Formatted at 2026-09-28 22:44:53 on dist-test-slave-pxr1"
server_key: "e8ac82f5bfa09c7d0d9b6a48afacd88a"
server_key_iv: "2a10dc9a6773505833ad30c9b9e01240"
server_key_version: "encryptionkey@0"
I20260928 22:44:53.854863 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "77a48f0a3d1043019b28f5921d5cfd98"
format_stamp: "Formatted at 2026-09-28 22:44:53 on dist-test-slave-pxr1"
server_key: "e8ac82f5bfa09c7d0d9b6a48afacd88a"
server_key_iv: "2a10dc9a6773505833ad30c9b9e01240"
server_key_version: "encryptionkey@0"
I20260928 22:44:53.856135 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:53.856902 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:53.857044 5933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:53.857127 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "77a48f0a3d1043019b28f5921d5cfd98"
format_stamp: "Formatted at 2026-09-28 22:44:53 on dist-test-slave-pxr1"
server_key: "e8ac82f5bfa09c7d0d9b6a48afacd88a"
server_key_iv: "2a10dc9a6773505833ad30c9b9e01240"
server_key_version: "encryptionkey@0"
I20260928 22:44:53.857191 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:53.870011 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:53.871739 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:53.871856 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:53.875514 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39567
I20260928 22:44:53.875555 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39567 every 8 connection(s)
I20260928 22:44:53.876016 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:53.876605 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:53.878780 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98: Bootstrap starting.
I20260928 22:44:53.879284 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:53.879516 6001 log.cc:824] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:53.880129 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98: No bootstrap required, opened a new log
I20260928 22:44:53.881640 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } }
I20260928 22:44:53.881775 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:53.881796 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a48f0a3d1043019b28f5921d5cfd98, State: Initialized, Role: FOLLOWER
I20260928 22:44:53.881896 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } }
I20260928 22:44:53.881950 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:53.881974 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:53.882002 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:53.882836 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } }
I20260928 22:44:53.882931 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77a48f0a3d1043019b28f5921d5cfd98; no voters:
I20260928 22:44:53.883064 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:53.883111 6004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:53.883260 6004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [term 1 LEADER]: Becoming Leader. State: Replica: 77a48f0a3d1043019b28f5921d5cfd98, State: Running, Role: LEADER
I20260928 22:44:53.883279 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:53.883364 6004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } }
I20260928 22:44:53.883725 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77a48f0a3d1043019b28f5921d5cfd98. Latest consensus state: current_term: 1 leader_uuid: "77a48f0a3d1043019b28f5921d5cfd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } } }
I20260928 22:44:53.883831 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:53.883723 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77a48f0a3d1043019b28f5921d5cfd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a48f0a3d1043019b28f5921d5cfd98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39567 } } }
I20260928 22:44:53.883942 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:53.884038 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:53.884526 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:53.885982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 5933
I20260928 22:44:53.886078 6013 catalog_manager.cc:1347] Generated new cluster ID: 2951dafe4dd841f5bdfc4358766b5e34
I20260928 22:44:53.886088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:53.886122 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:53.886288 392 external_mini_cluster.cc:1428] Setting key c286a8df958ab65727b140628586f2a0
Sep 28 22:44:53 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:53.894496 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:53.895114 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:53.897838 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:53.886718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.62:39567)
0928 22:44:53.886866 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:53.886875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:53.886910 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:44:53.887039 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:53.887046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:53.887101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:53.887242 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:53.887884 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:53.889330 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:53.890347 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:53.891042 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:53.891156 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:53.893826 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:53.893830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:53.893830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:53.893843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:53.895808 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:53.896212 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:53.896216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:53.896216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:53.896331 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:53.896521 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:53.896522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:53.896522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:53.896822 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:53.896970 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:53.897447 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 22:44:53.897602 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260928 22:44:53.908144 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77a48f0a3d1043019b28f5921d5cfd98: Generated new TSK 0
I20260928 22:44:53.908384 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:44:53.941550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:54.022814 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:54.023067 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:54.023096 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:54.024325 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:54.024369 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:54.024396 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:54.025949 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:54.026134 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:54.026314 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:44:54.031054 6027 server_base.cc:1034] running on GCE node
W20260928 22:44:54.031037 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.031037 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.031018 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:54.031442 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:54.031654 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:54.032796 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790635494032759 us; error 29 us; skew 500 ppm
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.035228 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:44:54.035650 6027 webserver.cc:460] Webserver started at http://127.0.98.1:33707/ using document root <none> and password file <none>
I20260928 22:44:54.035857 6027 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:54.035898 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:54.035989 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:54.037037 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a9c53ef71af400abe558e2196873745"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "feb23e8fd4ab095d3716c9adf94f1f1f"
server_key_iv: "f47775ac004edc364c443d5b6fd3a53c"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.037325 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a9c53ef71af400abe558e2196873745"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "feb23e8fd4ab095d3716c9adf94f1f1f"
server_key_iv: "f47775ac004edc364c443d5b6fd3a53c"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.038472 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:54.039151 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.039361 6027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:54.039439 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "6a9c53ef71af400abe558e2196873745"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "feb23e8fd4ab095d3716c9adf94f1f1f"
server_key_iv: "f47775ac004edc364c443d5b6fd3a53c"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.039492 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:54.052367 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:54.054132 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:54.054239 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:54.054425 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:54.054708 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:54.054739 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.054761 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:54.054783 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.060155 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44025
I20260928 22:44:54.060211 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44025 every 8 connection(s)
I20260928 22:44:54.060510 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:54.065960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6027
I20260928 22:44:54.066071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:54.066238 392 external_mini_cluster.cc:1428] Setting key d49814a5fe8123771d3ce387d3653535
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:54.069806 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.061233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39251 (local address 127.0.98.62:39567)
0928 22:44:54.061278 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.061279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.061477 (+ 198us) server_negotiation.cc:407] Connection header received
0928 22:44:54.062401 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.062402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.062416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.062475 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.063054 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.063545 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.064516 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.064707 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.064814 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.066938 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.066940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.066940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.066948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.068334 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.068826 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.068827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.068827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.068905 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.069180 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.069181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.069182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.069309 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.069372 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.069665 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.069733 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:54.070035 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.061206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39567 (local address 127.0.98.1:39251)
0928 22:44:54.061356 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:54.061390 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:54.062287 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:54.062532 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:54.062535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:54.062789 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:54.062980 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.062984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.063613 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.063614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.064452 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.064455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.064816 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.064817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.064919 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.065365 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:54.065374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:54.066861 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:54.068410 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.068418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.068421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.068741 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.068963 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.068964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.068964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.069101 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.069369 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:54.069373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:54.069532 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 22:44:54.069641 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:44:54.069805 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 22:44:54.070490 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39567
I20260928 22:44:54.070585 6156 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:54.070752 6156 heartbeater.cc:511] Master 127.0.98.62:39567 requested a full tablet report, sending...
I20260928 22:44:54.071296 5965 ts_manager.cc:194] Registered new tserver with Master: 6a9c53ef71af400abe558e2196873745 (127.0.98.1:44025)
I20260928 22:44:54.072191 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39251
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:44:54.089357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:54.169456 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:54.169677 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:54.169703 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:54.170871 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:54.170910 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:54.170935 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:54.172413 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:54.172621 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:54.172837 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:54.177639 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.177631 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.177631 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:54.177793 6161 server_base.cc:1034] running on GCE node
I20260928 22:44:54.177999 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:54.178231 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:54.179391 6161 hybrid_clock.cc:648] HybridClock initialized: now 1790635494179374 us; error 24 us; skew 500 ppm
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.181778 6161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:44:54.182226 6161 webserver.cc:460] Webserver started at http://127.0.98.2:33441/ using document root <none> and password file <none>
I20260928 22:44:54.182425 6161 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:54.182475 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:54.182585 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:54.183698 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "9bc70addfe66447aa1c0d642708e0776"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "c82d7fff817b7e1e4267639b126d42d3"
server_key_iv: "f6b335d66c45a489e423241c61ea4af0"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.184098 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9bc70addfe66447aa1c0d642708e0776"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "c82d7fff817b7e1e4267639b126d42d3"
server_key_iv: "f6b335d66c45a489e423241c61ea4af0"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.185482 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:54.186236 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.186399 6161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:54.186487 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "9bc70addfe66447aa1c0d642708e0776"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "c82d7fff817b7e1e4267639b126d42d3"
server_key_iv: "f6b335d66c45a489e423241c61ea4af0"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.186556 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:54.198071 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:54.199872 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:54.200004 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:54.200208 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:54.200489 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:54.200521 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.200551 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:54.200574 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.206127 6161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44973
I20260928 22:44:54.206166 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44973 every 8 connection(s)
I20260928 22:44:54.206477 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:54.213864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6161
I20260928 22:44:54.213979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:54.214150 392 external_mini_cluster.cc:1428] Setting key e20755d5ab515434684d49b1384768f9
I20260928 22:44:54.216084 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.207243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41263 (local address 127.0.98.62:39567)
0928 22:44:54.207302 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.207304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.207501 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:44:54.208613 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.208615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.208627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.208679 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.209421 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.209927 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.210913 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.211105 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.211160 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.213310 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.213312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.213312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.213320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.214702 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.215205 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.215206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.215207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.215270 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.215480 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.215481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.215481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.215606 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.215667 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.215914 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.215993 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:54.216523 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.207221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39567 (local address 127.0.98.2:41263)
0928 22:44:54.207382 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:54.207418 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:54.208469 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:54.208869 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:54.208873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:54.209129 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:54.209320 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.209325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.210012 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.210013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.210818 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.210820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.211171 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.211173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.211301 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.211792 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:54.211801 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:54.213209 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:54.214898 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.214904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.214906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.215135 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.215331 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.215331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.215332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.215432 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.215659 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:54.215663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:54.215802 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 22:44:54.215886 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:44:54.216022 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260928 22:44:54.217000 6290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39567
I20260928 22:44:54.217106 6290 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:54.217288 6290 heartbeater.cc:511] Master 127.0.98.62:39567 requested a full tablet report, sending...
I20260928 22:44:54.217638 5965 ts_manager.cc:194] Registered new tserver with Master: 9bc70addfe66447aa1c0d642708e0776 (127.0.98.2:44973)
I20260928 22:44:54.218125 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41263
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:44:54.237464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:54.318526 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:54.318737 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:54.318763 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:54.319885 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:54.319918 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:54.319950 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:54.321439 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:54.321628 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:54.321807 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:54.326383 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.326392 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:54.326392 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:54.326592 6295 server_base.cc:1034] running on GCE node
I20260928 22:44:54.326797 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:54.327028 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:54.328157 6295 hybrid_clock.cc:648] HybridClock initialized: now 1790635494328142 us; error 33 us; skew 500 ppm
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.330586 6295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:44:54.331053 6295 webserver.cc:460] Webserver started at http://127.0.98.3:44447/ using document root <none> and password file <none>
I20260928 22:44:54.331264 6295 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:54.331310 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:54.331427 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:54.332455 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "45de0729762d40e1a27d824ca7d84e2a"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "54602a71c08c1e2a0ef11373e589ff69"
server_key_iv: "0014feb52d3dbc56e7160909e8027c19"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.332836 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45de0729762d40e1a27d824ca7d84e2a"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "54602a71c08c1e2a0ef11373e589ff69"
server_key_iv: "0014feb52d3dbc56e7160909e8027c19"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.334071 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:54.334721 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.334874 6295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.334954 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "45de0729762d40e1a27d824ca7d84e2a"
format_stamp: "Formatted at 2026-09-28 22:44:54 on dist-test-slave-pxr1"
server_key: "54602a71c08c1e2a0ef11373e589ff69"
server_key_iv: "0014feb52d3dbc56e7160909e8027c19"
server_key_version: "encryptionkey@0"
I20260928 22:44:54.335018 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:54.347340 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:54.349200 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:54.349308 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:54.349488 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:54.349776 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:54.349807 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.349829 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:54.349851 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:54.355239 6295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45579
I20260928 22:44:54.355291 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45579 every 8 connection(s)
I20260928 22:44:54.355593 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:44:54.361981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6295
I20260928 22:44:54.362074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:54.362259 392 external_mini_cluster.cc:1428] Setting key 7e4a005beaa6340024db3959cfa3d543
I20260928 22:44:54.364854 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.356323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44449 (local address 127.0.98.62:39567)
0928 22:44:54.356388 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.356390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.356578 (+ 188us) server_negotiation.cc:407] Connection header received
0928 22:44:54.357580 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.357581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.357591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.357650 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.358243 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.358734 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.359767 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.359957 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.360003 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.362223 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.362225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.362225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.362235 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.363599 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.363981 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.363982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.363982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.364028 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.364240 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.364241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.364241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.364367 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.364433 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.364641 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.364702 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:44:54.365005 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.356287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39567 (local address 127.0.98.3:44449)
0928 22:44:54.356454 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:54.356497 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:54.357484 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:54.357712 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:54.357715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:54.357985 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:54.358178 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.358182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.358801 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.358801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.359708 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:54.359710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.360059 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.360060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:54.360189 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.360669 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:54.360676 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:54.362075 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:54.363679 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.363685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.363687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.363906 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.364069 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:54.364069 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:54.364070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:54.364161 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:54.364427 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:54.364430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:54.364542 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 22:44:54.364621 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:44:54.364782 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260928 22:44:54.365424 6424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39567
I20260928 22:44:54.365510 6424 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:54.365675 6424 heartbeater.cc:511] Master 127.0.98.62:39567 requested a full tablet report, sending...
I20260928 22:44:54.366039 5965 ts_manager.cc:194] Registered new tserver with Master: 45de0729762d40e1a27d824ca7d84e2a (127.0.98.3:45579)
I20260928 22:44:54.366393 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44449
I20260928 22:44:54.374119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/test-admin.keytab.
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.388080 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 22:44:54.389626 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:54.388939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.0.98.1:44025)
0928 22:44:54.389118 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.389124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.389161 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:44:54.389185 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.389187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.389201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.389257 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.389544 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59084: BlockingRecv error: recv got EOF from 127.0.0.1:59084 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
W20260928 22:44:54.389767 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.390784 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.391592 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:54.399143 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.392099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.98.1:44025)
0928 22:44:54.392147 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.392149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.392157 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:54.392230 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.392231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.392243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.392290 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.392658 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.393455 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.394169 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.394508 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.394568 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.396512 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.396521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.396522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.396531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.398082 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.398408 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.398411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.398411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.398463 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.398621 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.398621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.398622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.398759 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.398897 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.398965 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.399040 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 22:44:54.400703 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:54.400244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.0.98.1:44025)
0928 22:44:54.400299 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.400301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.400309 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:54.400372 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.400373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.400384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.400422 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.400659 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59108: BlockingRecv error: recv got EOF from 127.0.0.1:59108 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 22:44:54.400837 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.401698 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.402477 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:54.408787 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.402929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.0.98.1:44025)
0928 22:44:54.402974 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.402976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.403034 (+ 58us) server_negotiation.cc:407] Connection header received
0928 22:44:54.403088 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.403089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.403099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.403137 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.403474 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.403942 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.404588 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.404818 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.404871 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.406712 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.406713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.406713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.406720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.407777 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.408093 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.408094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.408094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.408146 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.408298 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.408299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.408299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.408396 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.408484 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.408592 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.408670 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 22:44:54.410130 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:54.409659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.1:44025)
0928 22:44:54.409713 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.409715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.409726 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:54.409781 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.409782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.409791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.409841 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.410085 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59120: BlockingRecv error: recv got EOF from 127.0.0.1:59120 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 22:44:54.410175 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635468897675-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.411065 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:54.411857 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:44:54.418316 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:54.412291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.0.98.1:44025)
0928 22:44:54.412323 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:54.412325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:54.412369 (+ 44us) server_negotiation.cc:407] Connection header received
0928 22:44:54.412421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:54.412421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:54.412428 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:54.412473 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:54.412848 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.413312 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.413961 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:54.414135 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:54.414187 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:54.416130 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:54.416132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:54.416132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:54.416140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:54.417333 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.417640 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.417641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.417642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.417689 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:54.417875 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:54.417876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:54.417876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:54.417969 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:54.418049 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:54.418150 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:44:54.418223 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:44:54.418946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6027
I20260928 22:44:54.423513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6161
I20260928 22:44:54.427121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6295
I20260928 22:44:54.430539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 5933
2026-09-28T22:44:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1726 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[6444](info): set up 2 sockets
Sep 28 22:44:54 dist-test-slave-pxr1 krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260928 22:44:55.376672 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:44:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 22:44:55.483208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:55.561579 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:55.561787 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:55.561817 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:55.562947 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:55.562986 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:55.563004 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:55.563020 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:55.563035 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:55.564457 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:55.564711 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:55.564932 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:55.569429 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:55.569446 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:55.569440 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:55.569733 6457 server_base.cc:1034] running on GCE node
I20260928 22:44:55.569945 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:55.570183 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:55.571316 6457 hybrid_clock.cc:648] HybridClock initialized: now 1790635495571305 us; error 30 us; skew 500 ppm
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:55.573748 6457 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 22:44:55.574162 6457 webserver.cc:460] Webserver started at http://127.0.98.62:44509/ using document root <none> and password file <none>
I20260928 22:44:55.574340 6457 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:55.574378 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:55.574467 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:55.575438 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "47571772bac54152978e413c10de5fe3"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "f8bb0d126e264c8db7bc98dda450d647"
server_key_iv: "79d0bdf93ebb74c0cfdb3c51d76897e6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.575730 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "47571772bac54152978e413c10de5fe3"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "f8bb0d126e264c8db7bc98dda450d647"
server_key_iv: "79d0bdf93ebb74c0cfdb3c51d76897e6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.576891 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:44:55.577520 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.577668 6457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:55.577770 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "47571772bac54152978e413c10de5fe3"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "f8bb0d126e264c8db7bc98dda450d647"
server_key_iv: "79d0bdf93ebb74c0cfdb3c51d76897e6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.577828 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:55.585181 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:55.587229 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:55.587354 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:55.590953 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45071
I20260928 22:44:55.590996 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45071 every 8 connection(s)
I20260928 22:44:55.591323 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:44:55.592247 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:55.594568 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Bootstrap starting.
I20260928 22:44:55.595127 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:55.595427 6525 log.cc:824] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:55.596081 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: No bootstrap required, opened a new log
I20260928 22:44:55.597520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6457
I20260928 22:44:55.597626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:44:55.597819 392 external_mini_cluster.cc:1428] Setting key d2912738440c66a79d96b2f78e7afc6d
I20260928 22:44:55.597906 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260928 22:44:55.598071 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:55.598095 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47571772bac54152978e413c10de5fe3, State: Initialized, Role: FOLLOWER
I20260928 22:44:55.598235 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260928 22:44:55.598309 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:44:55.598340 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:44:55.598378 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:55.598906 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260928 22:44:55.598997 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47571772bac54152978e413c10de5fe3; no voters:
I20260928 22:44:55.599146 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:44:55.599177 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:55.599280 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 47571772bac54152978e413c10de5fe3, State: Running, Role: LEADER
I20260928 22:44:55.599370 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260928 22:44:55.599408 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:55.599692 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } }
I20260928 22:44:55.599767 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:55.599718 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47571772bac54152978e413c10de5fe3. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } }
I20260928 22:44:55.599824 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:55.600013 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:55.600540 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:55.602712 6536 catalog_manager.cc:1347] Generated new cluster ID: d9846d636e0b498bba87cf6c1b393722
I20260928 22:44:55.602761 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:55.610338 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:55.598352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.98.62:45071)
0928 22:44:55.598518 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:55.598529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:55.598562 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:44:55.598903 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:55.598909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:55.598953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:55.599095 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:55.600088 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.601610 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.602393 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.603015 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.603126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:55.605986 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:55.605991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:55.605991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:55.606006 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:55.608500 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.608962 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.608964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.608965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.609090 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.609281 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.609282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.609282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.609667 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:55.609841 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:55.609964 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 22:44:55.610108 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260928 22:44:55.630059 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:44:55.630577 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:55.632323 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Generated new TSK 0
I20260928 22:44:55.632510 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 22:44:55.665006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:55.748626 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:55.748848 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:55.748870 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:55.750087 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:55.750128 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:55.750152 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:55.751525 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:55.751713 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:55.751888 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:55.756472 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:55.756634 6551 server_base.cc:1034] running on GCE node
W20260928 22:44:55.756487 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:55.756459 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:55.756959 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:55.757162 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:55.758293 6551 hybrid_clock.cc:648] HybridClock initialized: now 1790635495758279 us; error 26 us; skew 500 ppm
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:55.760576 6551 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 22:44:55.761036 6551 webserver.cc:460] Webserver started at http://127.0.98.1:41269/ using document root <none> and password file <none>
I20260928 22:44:55.761227 6551 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:55.761274 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:55.761394 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:55.762396 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "dad25693f8c04a6997fa58bf2991aaab"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "5cd7b2ab6d89c5cf1c57104a754bf533"
server_key_iv: "2a9d5fa29dc86c02637db9c16e80d5fb"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.762729 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dad25693f8c04a6997fa58bf2991aaab"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "5cd7b2ab6d89c5cf1c57104a754bf533"
server_key_iv: "2a9d5fa29dc86c02637db9c16e80d5fb"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.763926 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:44:55.764541 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.764710 6551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:55.764825 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "dad25693f8c04a6997fa58bf2991aaab"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "5cd7b2ab6d89c5cf1c57104a754bf533"
server_key_iv: "2a9d5fa29dc86c02637db9c16e80d5fb"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.764889 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:55.789180 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:55.791023 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:55.791163 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:55.791392 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:55.791703 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:55.791736 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.791767 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:55.791788 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.797156 6551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46165
I20260928 22:44:55.797197 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46165 every 8 connection(s)
I20260928 22:44:55.797513 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:55.799958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6551
I20260928 22:44:55.800060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:44:55.800282 392 external_mini_cluster.cc:1428] Setting key 76fd988147a3efe5367d3a605f61df19
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:55.806908 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:55.798219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39927 (local address 127.0.98.62:45071)
0928 22:44:55.798283 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:55.798285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:55.798510 (+ 225us) server_negotiation.cc:407] Connection header received
0928 22:44:55.798963 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:55.798964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:55.798978 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:55.799027 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:55.800064 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.800820 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.801837 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.802046 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.802101 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:55.804301 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:55.804303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:55.804303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:55.804311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:55.805628 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.806021 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.806022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.806022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.806077 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.806291 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.806292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.806293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.806405 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:55.806460 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:55.806725 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 22:44:55.806818 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:44:55.807075 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:55.798181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:39927)
0928 22:44:55.798350 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:55.798396 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:55.798886 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:55.799093 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:55.799097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:55.799687 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:55.799950 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:55.799956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.800910 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.800912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:55.801746 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:55.801748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.802118 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.802119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:55.802253 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:55.802757 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:55.802766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:55.804207 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:55.805713 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:55.805738 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:55.805740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:55.805954 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:55.806130 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:55.806131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:55.806131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:55.806239 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:55.806459 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:55.806465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:55.806604 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 22:44:55.806694 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:44:55.806857 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:44:55.807479 6680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:55.807574 6680 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:55.807739 6680 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:55.808220 6489 ts_manager.cc:194] Registered new tserver with Master: dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
I20260928 22:44:55.808892 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 22:44:55.823220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:55.901894 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:55.902107 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:55.902138 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:55.903303 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:55.903348 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:55.903388 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:55.904870 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:55.905103 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:55.905308 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:55.909927 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:55.909977 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:55.910140 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:55.910132 6685 server_base.cc:1034] running on GCE node
I20260928 22:44:55.910336 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:55.910544 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:55.911700 6685 hybrid_clock.cc:648] HybridClock initialized: now 1790635495911680 us; error 31 us; skew 500 ppm
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:55.914129 6685 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 22:44:55.914547 6685 webserver.cc:460] Webserver started at http://127.0.98.2:46877/ using document root <none> and password file <none>
I20260928 22:44:55.914732 6685 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:55.914770 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:55.914873 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:55.915889 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "59c6a5659bf64cef8715f0d82c50175a"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "26b2acf003b6d83c418ab2683189916f"
server_key_iv: "177c0e4ecbb5719ebeafc0d5495a73b6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.916183 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59c6a5659bf64cef8715f0d82c50175a"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "26b2acf003b6d83c418ab2683189916f"
server_key_iv: "177c0e4ecbb5719ebeafc0d5495a73b6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.917433 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:55.918071 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.918243 6685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:55.918323 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "59c6a5659bf64cef8715f0d82c50175a"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "26b2acf003b6d83c418ab2683189916f"
server_key_iv: "177c0e4ecbb5719ebeafc0d5495a73b6"
server_key_version: "encryptionkey@0"
I20260928 22:44:55.918380 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:55.936250 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:55.938103 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:55.938225 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:55.938410 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:55.938704 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:55.938735 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.938766 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:55.938788 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:55.944316 6685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43117
I20260928 22:44:55.944357 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43117 every 8 connection(s)
I20260928 22:44:55.944669 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:55.947517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6685
I20260928 22:44:55.947608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:44:55.947799 392 external_mini_cluster.cc:1428] Setting key 0c9886da299cf2166ba098421ba3bb45
Sep 28 22:44:55 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:55.954468 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:55.945495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50967 (local address 127.0.98.62:45071)
0928 22:44:55.945570 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:55.945572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:55.945759 (+ 187us) server_negotiation.cc:407] Connection header received
0928 22:44:55.946255 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:55.946256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:55.946271 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:55.946333 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:55.947240 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.947782 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.948937 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.949134 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.949185 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:55.951442 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:55.951443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:55.951444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:55.951451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:55.953077 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.953584 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.953585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.953585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.953650 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:55.953864 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:55.953864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:55.953865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:55.954002 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:55.954064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:55.954290 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 22:44:55.954376 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:44:55.954677 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:55.945426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:50967)
0928 22:44:55.945599 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:55.945649 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:55.946143 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:55.946395 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:55.946400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:55.946971 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:55.947170 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:55.947175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.947869 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.947870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:55.948854 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:55.948857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:55.949198 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:55.949199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:55.949318 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:55.949889 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:55.949899 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:55.951354 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:55.953290 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:55.953295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:55.953297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:55.953509 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:55.953713 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:55.953713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:55.953713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:55.953809 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:55.954061 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:55.954064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:55.954178 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 22:44:55.954251 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:44:55.954376 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 22:44:55.955227 6814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:55.955315 6814 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:55.955484 6814 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:55.955850 6489 ts_manager.cc:194] Registered new tserver with Master: 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 22:44:55.956202 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 22:44:55.971426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:56.049719 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.049940 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.049971 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.051132 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.051174 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.051209 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:56.052632 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:56.052902 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.053114 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.057777 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.057832 6819 server_base.cc:1034] running on GCE node
W20260928 22:44:56.057750 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.057750 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.058158 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.058362 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.059491 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790635496059478 us; error 29 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.061869 6819 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 22:44:56.062300 6819 webserver.cc:460] Webserver started at http://127.0.98.3:34743/ using document root <none> and password file <none>
I20260928 22:44:56.062476 6819 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.062516 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.062602 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:56.063587 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee24594ebeba400aa5a000075698796b"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "3c5f77a1fee93d73f860805f96f1b1b9"
server_key_iv: "5ea119007dadcf5c15d946c37422e233"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.063885 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee24594ebeba400aa5a000075698796b"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "3c5f77a1fee93d73f860805f96f1b1b9"
server_key_iv: "5ea119007dadcf5c15d946c37422e233"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.065114 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:44:56.065774 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.065922 6819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:56.065999 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "ee24594ebeba400aa5a000075698796b"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "3c5f77a1fee93d73f860805f96f1b1b9"
server_key_iv: "5ea119007dadcf5c15d946c37422e233"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.066064 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.088876 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.090875 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.090983 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.091164 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:56.091454 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:56.091487 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.091516 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:56.091542 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.096935 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39077
I20260928 22:44:56.096987 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39077 every 8 connection(s)
I20260928 22:44:56.097285 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:56.105942 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.097971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51131 (local address 127.0.98.62:45071)
0928 22:44:56.098044 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.098046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.098214 (+ 168us) server_negotiation.cc:407] Connection header received
0928 22:44:56.098682 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.098683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.098716 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.098778 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.099769 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.100293 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.101337 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.101525 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.101580 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.103550 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.103551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.103551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.103558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.104671 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.105058 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.105060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.105060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.105113 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.105326 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.105327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.105327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.105434 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.105489 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.105761 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.105849 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:44:56.105978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6819
I20260928 22:44:56.106084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:44:56.106112 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.097945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:51131)
0928 22:44:56.098091 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.098135 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.098601 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.098837 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.098840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.099484 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.099689 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.099695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.100402 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.100403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.101264 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.101267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.101586 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.101587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.101684 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.102178 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.102186 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.103462 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.104764 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.104771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.104773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.104991 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.105167 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.105168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.105168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.105267 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.105489 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.105496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.105641 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.105731 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.105893 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 22:44:56.106281 392 external_mini_cluster.cc:1428] Setting key 16755d8bd4c31759d24aaa75bcdb9b93
I20260928 22:44:56.106593 6948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:56.106698 6948 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.106868 6948 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:56.107234 6489 ts_manager.cc:194] Registered new tserver with Master: ee24594ebeba400aa5a000075698796b (127.0.98.3:39077)
I20260928 22:44:56.107595 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51131
I20260928 22:44:56.117866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 22:44:56.140539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34473
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45071,127.0.98.61:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:56.218425 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.218634 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.218664 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.219815 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:56.219861 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.219878 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.219894 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:56.219909 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:56.221431 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45071,127.0.98.61:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
W20260928 22:44:56.221544 6953 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45071,127.0.98.61:34473'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 22:44:56.221684 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.221885 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.226426 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.226455 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.226426 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.226612 6953 server_base.cc:1034] running on GCE node
I20260928 22:44:56.227101 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.227403 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.228551 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790635496228523 us; error 46 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.231102 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 22:44:56.231503 6953 webserver.cc:460] Webserver started at http://127.0.98.61:37681/ using document root <none> and password file <none>
I20260928 22:44:56.231681 6953 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.231720 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.231808 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:56.232856 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/instance:
uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "55d8401a8187658d1e58e9d12af9ace8"
server_key_iv: "e063aa04b4a6c6eb62eaeab3a36e8111"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.233151 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal/instance:
uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "55d8401a8187658d1e58e9d12af9ace8"
server_key_iv: "e063aa04b4a6c6eb62eaeab3a36e8111"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.234337 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:56.235062 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.235251 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:44:56.235335 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "55d8401a8187658d1e58e9d12af9ace8"
server_key_iv: "e063aa04b4a6c6eb62eaeab3a36e8111"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.235402 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.247934 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.249784 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.249912 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.253387 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34473
I20260928 22:44:56.253420 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34473 every 8 connection(s)
I20260928 22:44:56.253757 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
I20260928 22:44:56.254563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 6953
I20260928 22:44:56.254784 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:56.257040 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:56.264518 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.257734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.0.98.62:45071)
0928 22:44:56.257796 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.257798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.257930 (+ 132us) server_negotiation.cc:407] Connection header received
0928 22:44:56.257995 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.257996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.258006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.258059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.258439 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.259063 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.260037 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.260332 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.260408 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.262215 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.262217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.262217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.262225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.263481 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263799 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.263801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.263801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.263850 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263998 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.263998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.263999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.264132 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.264195 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.264353 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.264429 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:44:56.264570 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.255118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.61:34473)
0928 22:44:56.255300 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.255309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.255352 (+ 43us) server_negotiation.cc:407] Connection header received
0928 22:44:56.255945 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.255954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.256020 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.256176 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.256707 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.257947 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.258810 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.259247 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.259342 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.261656 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.261663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.261663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.261670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.263226 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263529 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.263532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.263532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.263612 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263779 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.263780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.263781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.263996 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.264112 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.264238 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.264351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 22:44:56.264588 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.257669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36080)
0928 22:44:56.257826 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.257865 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.257937 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.258075 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.258078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.258313 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.258372 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.258373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.259157 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.259158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.259957 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.259959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.260409 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.260410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.260458 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.260897 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.260905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.262135 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.263551 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.263560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.263738 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.263901 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.263901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.263902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.263951 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.264195 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.264201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.264257 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.264323 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.264502 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":107,"threads_started":1}
I20260928 22:44:56.265115 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 22:44:56.270387 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.265340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43664)
0928 22:44:56.265382 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.265392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.265421 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.265519 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.265520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.265655 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.265694 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.265695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.266256 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.266256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.266716 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.266717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.267082 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.267083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.267129 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.267411 (+ 282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.267414 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.268466 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.269618 (+ 1152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.269619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.269619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.269832 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.269966 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.269966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.269966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.270009 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.270166 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.270167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.270197 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.270239 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.270306 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:44:56.270430 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.265374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.61:34473)
0928 22:44:56.265404 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.265406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.265424 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:44:56.265472 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.265472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.265480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.265519 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.265754 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.266194 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.266777 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.266968 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.267043 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.268535 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.268536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.268537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.268542 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.269551 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.269878 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.269878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.269879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.269921 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.270044 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.270044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.270045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.270125 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.270174 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.270255 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.270345 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":18}
I20260928 22:44:56.271588 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Bootstrap starting.
I20260928 22:44:56.272346 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:56.272588 7021 log.cc:824] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:56.273289 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: No bootstrap required, opened a new log
I20260928 22:44:56.274622 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } }
I20260928 22:44:56.274744 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:56.274768 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f52be26196463c9f6c89eb1cf7bb5d, State: Initialized, Role: FOLLOWER
I20260928 22:44:56.274880 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } }
I20260928 22:44:56.275120 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } } }
I20260928 22:44:56.275173 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:56.275198 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:44:56.276579 7038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:44:56.276633 7038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:44:56.278090 6953 master_runner.cc:428] Detected that this master 20f52be26196463c9f6c89eb1cf7bb5d is joining an existing cluster
I20260928 22:44:56.278126 6953 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34473
I20260928 22:44:56.285231 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.278988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.0.98.61:34473)
0928 22:44:56.279022 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.279024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.279246 (+ 222us) server_negotiation.cc:407] Connection header received
0928 22:44:56.279302 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.279303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.279310 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.279349 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.279714 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.280439 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281421 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.281686 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281734 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.282691 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.282692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.282692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.282698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.284155 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284522 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.284523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.284523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.284572 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284762 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.284762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.284763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.284878 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.284909 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.285065 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.285147 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:44:56.285449 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.279180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.98.62:45071)
0928 22:44:56.279225 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.279227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.279234 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:56.279255 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.279256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.279264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.279307 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.279669 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.280412 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281345 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.281634 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281697 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.282806 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.282807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.282808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.282814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.284254 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284689 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.284690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.284690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.284770 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284952 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.284953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.284953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.285074 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.285134 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.285294 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.285366 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:56.285524 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.278962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36088)
0928 22:44:56.279103 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.279112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.279159 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.279347 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.279348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.279534 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.279598 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.279600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.280501 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.280502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.281274 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.281277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.281702 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.281765 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.282328 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.282332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.282719 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.284346 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.284347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.284607 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.284824 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.284826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.284892 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.285145 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.285146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.285190 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.285259 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.285419 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260928 22:44:56.285898 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.279013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43678)
0928 22:44:56.279191 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.279200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.279260 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.279364 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.279364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.279588 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.279657 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.279659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.280508 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.280508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.281325 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.281327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.281772 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.281772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.281820 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.282254 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.282259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.282610 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.284240 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.284241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.284464 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.284623 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.284624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.284624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.284676 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.284941 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.284942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.284975 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.285310 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.285492 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":92,"thread_start_us":62,"threads_started":1}
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 22:44:56.294796 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.287594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43686)
0928 22:44:56.287710 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.287750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.287815 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.287947 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.287949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.288185 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.288233 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.288234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.289130 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.289131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.289731 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.289732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.290188 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.290189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.290283 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.290741 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.290745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.292235 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.293850 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.293857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.293859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.294051 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.294236 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.294236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.294237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.294294 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.294525 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.294529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.294571 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.294630 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.294722 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260928 22:44:56.294838 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.287642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.0.98.61:34473)
0928 22:44:56.287681 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.287683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.287810 (+ 127us) server_negotiation.cc:407] Connection header received
0928 22:44:56.287870 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.287870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.287879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.287942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.288292 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.289051 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.289808 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.290103 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.290174 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.292322 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.292323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.292324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.292332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.293760 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.294121 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.294122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.294122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.294178 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.294344 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.294344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.294345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.294469 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.294526 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.294668 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.294740 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:56.295339 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34473: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34473
}
attrs {
promote: true
}
}
dest_uuid: "47571772bac54152978e413c10de5fe3"
cas_config_opid_index: -1
I20260928 22:44:56.295626 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } }
I20260928 22:44:56.296051 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } }
I20260928 22:44:56.296442 6995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:56.297135 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34473
I20260928 22:44:56.297326 6995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 22:44:56.297355 6953 master.cc:537] Master@127.0.98.61:34473 shutting down...
I20260928 22:44:56.297339 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } }
I20260928 22:44:56.297400 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:56.297621 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:44:56.298875 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 22:44:56.299292 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 -> Peer 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61:34473): Couldn't send request to peer 20f52be26196463c9f6c89eb1cf7bb5d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:44:56.300173 6953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:56.300267 6953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:56.300316 6953 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: stopping tablet replica
I20260928 22:44:56.312824 6953 master.cc:552] Master@127.0.98.61:34473 shutdown complete.
I20260928 22:44:56.312873 6953 master_runner.cc:305] Clearing existing system tablet
I20260928 22:44:56.312991 6953 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.313031 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.313599 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:56.313999 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.314142 6953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.314193 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.314214 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "55d8401a8187658d1e58e9d12af9ace8"
server_key_iv: "e063aa04b4a6c6eb62eaeab3a36e8111"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.315222 6953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:56.316807 6953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:44:56.316864 6953 log.cc:1196] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 22:44:56.317026 6953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Deleting consensus metadata
I20260928 22:44:56.317121 6953 master_runner.cc:315] Copying system tablet from 127.0.98.62:45071
I20260928 22:44:56.317668 6953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45071
I20260928 22:44:56.323190 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.317997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.98.62:45071)
0928 22:44:56.318039 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.318041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.318139 (+ 98us) server_negotiation.cc:407] Connection header received
0928 22:44:56.318203 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.318204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.318214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.318267 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.318570 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.319067 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.319778 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.319972 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.320023 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.320963 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.320965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.320965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.320972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.322238 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.322525 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.322527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.322527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.322580 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.322733 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.322734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.322734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.322848 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.322902 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.323032 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.323099 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:44:56.323207 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.317942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36098)
0928 22:44:56.318058 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.318076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.318144 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.318272 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.318273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.318453 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.318500 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.318502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.319139 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.319140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.319706 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.319708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.320028 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.320028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.320087 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.320583 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.320588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.320888 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.322314 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.322315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.322316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.322467 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.322631 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.322631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.322632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.322684 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.322900 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.322900 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.322944 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.323004 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.323099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260928 22:44:56.323671 6509 tablet_copy_service.cc:140] P 47571772bac54152978e413c10de5fe3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20f52be26196463c9f6c89eb1cf7bb5d ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36098)
I20260928 22:44:56.323750 6509 tablet_copy_service.cc:161] P 47571772bac54152978e413c10de5fe3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20f52be26196463c9f6c89eb1cf7bb5d at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36098: session id = 20f52be26196463c9f6c89eb1cf7bb5d-00000000000000000000000000000000
I20260928 22:44:56.324391 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:44:56.325115 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:44:56.326385 6953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: tablet copy: Starting download of 0 data blocks...
I20260928 22:44:56.326537 6953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: tablet copy: Starting download of 1 WAL segments...
I20260928 22:44:56.327353 6953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:44:56.328243 6509 tablet_copy_service.cc:342] P 47571772bac54152978e413c10de5fe3: Request end of tablet copy session 20f52be26196463c9f6c89eb1cf7bb5d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36098
I20260928 22:44:56.328298 6509 tablet_copy_service.cc:433] P 47571772bac54152978e413c10de5fe3: ending tablet copy session 20f52be26196463c9f6c89eb1cf7bb5d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20f52be26196463c9f6c89eb1cf7bb5d
I20260928 22:44:56.330466 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.331454 7059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.331486 7060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.331537 7062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.331678 6953 server_base.cc:1034] running on GCE node
I20260928 22:44:56.331761 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.331893 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.332998 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790635496332989 us; error 29 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.334337 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 22:44:56.334592 6953 webserver.cc:460] Webserver started at http://127.0.98.61:37079/ using document root <none> and password file <none>
I20260928 22:44:56.334703 6953 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.334736 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.335398 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:56.335906 7069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.336025 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.336057 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "55d8401a8187658d1e58e9d12af9ace8"
server_key_iv: "e063aa04b4a6c6eb62eaeab3a36e8111"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.336112 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.341672 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.343031 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.345428 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34473
I20260928 22:44:56.345798 7121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34473 every 8 connection(s)
I20260928 22:44:56.345819 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
I20260928 22:44:56.346467 7122 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:44:56.346861 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Bootstrap starting.
I20260928 22:44:56.349300 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:44:56.349501 7122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: Bootstrap complete.
I20260928 22:44:56.349722 7122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } }
I20260928 22:44:56.349792 7122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20f52be26196463c9f6c89eb1cf7bb5d, State: Initialized, Role: LEARNER
I20260928 22:44:56.349817 7122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } }
I20260928 22:44:56.349977 7122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:56.349997 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } }
I20260928 22:44:56.350116 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: LEARNER
I20260928 22:44:56.351212 7136 catalog_manager.cc:1259] Loaded cluster ID: d9846d636e0b498bba87cf6c1b393722
I20260928 22:44:56.351253 7136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: loading cluster ID for follower catalog manager: success
I20260928 22:44:56.352144 7136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: acquiring CA information for follower catalog manager: success
I20260928 22:44:56.352389 7136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:44:56.372925 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.366015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.98.61:34473)
0928 22:44:56.366138 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.366140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.366150 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:56.366178 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.366179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.366190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.366289 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.366795 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.367916 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.368692 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.369054 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.369136 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.370312 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.370314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.370314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.370321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.371853 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.372158 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.372159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.372160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.372232 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.372388 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.372388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.372389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.372522 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.372617 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.372698 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.372815 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 22:44:56.395164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42803
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:44:56.474603 7140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.474820 7140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.474841 7140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.476020 7140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:44:56.476065 7140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.476078 7140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.476090 7140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:44:56.476112 7140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:44:56.477739 7140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:56.477977 7140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.478190 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.482790 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.482813 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.482803 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.482846 7140 server_base.cc:1034] running on GCE node
I20260928 22:44:56.483183 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.483465 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.484632 7140 hybrid_clock.cc:648] HybridClock initialized: now 1790635496484620 us; error 28 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.487284 7140 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 22:44:56.487758 7140 webserver.cc:460] Webserver started at http://127.0.98.60:32847/ using document root <none> and password file <none>
I20260928 22:44:56.487938 7140 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.487978 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.488070 7140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:44:56.489126 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/instance:
uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "ca1af97e8b7bec48b0d44d9aba384ad2"
server_key_iv: "df6b6b031d7ab6b7f77a9ed091d780e3"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.489450 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal/instance:
uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "ca1af97e8b7bec48b0d44d9aba384ad2"
server_key_iv: "df6b6b031d7ab6b7f77a9ed091d780e3"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.490593 7140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:44:56.491245 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.491376 7140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:56.491457 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "ca1af97e8b7bec48b0d44d9aba384ad2"
server_key_iv: "df6b6b031d7ab6b7f77a9ed091d780e3"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.491518 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.520291 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.522426 7140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.522565 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.526705 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42803
I20260928 22:44:56.526741 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42803 every 8 connection(s)
I20260928 22:44:56.527073 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
I20260928 22:44:56.528024 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:56.529582 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:44:56.529903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7140
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 22:44:56.538650 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.530408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.62:45071)
0928 22:44:56.530450 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.530451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.530653 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:44:56.531064 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.531066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.531078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.531126 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.531667 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.532158 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.533181 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.533376 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.533433 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.535492 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.535495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.535495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.535501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.537117 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.537546 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.537548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.537548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.537610 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.537812 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.537813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.537814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.537948 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.538015 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.538482 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.538560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:44:56.538763 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.530275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36110)
0928 22:44:56.530521 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.530564 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.530980 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.531153 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.531155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.531418 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.531573 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.531577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.532245 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.532246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.533097 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.533100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.533442 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.533443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.533507 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.534072 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.534080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.535413 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.537196 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.537203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.537205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.537480 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.537670 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.537671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.537671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.537758 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.538213 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.538218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.538344 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.538451 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.538571 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260928 22:44:56.538962 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.530396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.98.60:42803)
0928 22:44:56.530533 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.530542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.530564 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:44:56.530967 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.530974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.530993 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.531081 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.531638 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.532410 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.533099 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.533402 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.533489 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.535948 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.535955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.535956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.535962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.537944 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.538206 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.538209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.538209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.538260 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.538454 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.538454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.538455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.538597 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.538724 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.538807 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.538873 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260928 22:44:56.539230 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 22:44:56.544569 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.539455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43716)
0928 22:44:56.539559 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.539569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.539611 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.539716 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.539716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.539817 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.539851 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.539852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.540393 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.540393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.540926 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.540928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.541218 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.541218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.541259 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.541560 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.541563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.542559 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.543742 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.543743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.543743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.543888 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.544045 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.544046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.544046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.544098 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.544299 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.544300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.544337 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.544397 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.544480 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 22:44:56.544579 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.539515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.98.61:34473)
0928 22:44:56.539575 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.539576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.539600 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:44:56.539654 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.539655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.539665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.539709 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.539891 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.540345 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.540988 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.541173 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.541216 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.542640 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.542642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.542642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.542649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.543673 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.543946 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.543946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.543947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.543995 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.544145 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.544146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.544146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.544246 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.544298 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.544424 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.544498 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:44:56.545017 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 22:44:56.550225 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.545233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41076)
0928 22:44:56.545271 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.545280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.545316 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.545404 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.545404 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.545517 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.545540 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.545541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.546114 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.546115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.546567 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.546568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.546885 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.546886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.546928 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.547217 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.547221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.548197 (+ 976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.549377 (+ 1180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.549377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.549378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.549515 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.549660 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.549660 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.549661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.549737 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.549935 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.549936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.549969 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.550027 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.550119 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:44:56.550292 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.545271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.0.98.60:42803)
0928 22:44:56.545301 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.545303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.545310 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:56.545355 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.545356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.545363 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.545403 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.545605 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.546052 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.546630 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.546832 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.546883 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.548266 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.548267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.548267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.548273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.549313 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.549568 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.549569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.549570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.549611 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.549793 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.549794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.549794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.549886 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.549934 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.550054 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.550163 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":17}
I20260928 22:44:56.551542 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Bootstrap starting.
I20260928 22:44:56.552333 7208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:56.552582 7208 log.cc:824] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:56.553278 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: No bootstrap required, opened a new log
I20260928 22:44:56.554672 7208 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } }
I20260928 22:44:56.554803 7208 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:56.554826 7208 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecc4c8c3bcf4d55b97bc5190a6f62d7, State: Initialized, Role: FOLLOWER
I20260928 22:44:56.554944 7208 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } }
I20260928 22:44:56.555192 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } } }
I20260928 22:44:56.555240 7208 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:56.555286 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:44:56.556692 7224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:44:56.556782 7224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:44:56.558849 7140 master_runner.cc:428] Detected that this master aecc4c8c3bcf4d55b97bc5190a6f62d7 is joining an existing cluster
I20260928 22:44:56.558893 7140 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42803
I20260928 22:44:56.567265 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.98.61:34473)
0928 22:44:56.560332 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.560333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.560893 (+ 560us) server_negotiation.cc:407] Connection header received
0928 22:44:56.560938 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.560939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.560950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.560997 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.561425 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.562194 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563230 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563523 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563564 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.564936 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.564938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.564938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.564945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.566228 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.566529 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.566530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.566530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.566583 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.566803 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.566804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.566804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.566912 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.566945 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.567108 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.567179 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:44:56.567274 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43720)
0928 22:44:56.560829 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.560839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.560890 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.561051 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.561051 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.561280 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.561338 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.561340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.562297 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.562298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563134 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.563137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563642 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563696 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.564452 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.564456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.564856 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.566300 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.566301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.566301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.566469 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.566636 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.566636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.566637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.566696 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.566978 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.566979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.567013 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.567062 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.567163 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":21,"thread_start_us":60,"threads_started":1}
I20260928 22:44:56.584379 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41080)
0928 22:44:56.560965 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.560972 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.561026 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.561178 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.561179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.561337 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.561378 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.561379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.562237 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.562237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563207 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.563209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563638 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563691 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.581313 (+ 17622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.581320 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.581710 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.583323 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.583325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.583566 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.583757 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.583758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.583803 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.584016 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.584017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.584062 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.584122 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.584228 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":50,"threads_started":1}
I20260928 22:44:56.584509 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.98.62:45071)
0928 22:44:56.560131 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.560133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.560633 (+ 500us) server_negotiation.cc:407] Connection header received
0928 22:44:56.560661 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.560662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.560671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.560751 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.561132 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.561907 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563472 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563777 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563818 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.581917 (+ 18099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.581919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.581919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.581928 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.583386 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583738 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.583739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.583739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.583798 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583988 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.583989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.583990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.584130 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.584188 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.584354 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.584429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:44:56.584394 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.60:42803)
0928 22:44:56.560471 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.560472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.561009 (+ 537us) server_negotiation.cc:407] Connection header received
0928 22:44:56.561079 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.561080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.561087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.561123 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.561442 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.562157 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563273 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563533 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563564 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.581802 (+ 18238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.581803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.581803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.581810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.583232 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583638 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.583639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.583640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.583688 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583849 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.583850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.583850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.583962 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.584017 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.584163 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.584287 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":16}
I20260928 22:44:56.584826 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.560020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36124)
0928 22:44:56.560530 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.560540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.560596 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.560760 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.560761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.560969 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.561035 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.561037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.562000 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.562001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563362 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.563365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.563892 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.563892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.563952 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.581359 (+ 17407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.581382 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.581822 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.583447 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.583448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.583658 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.583869 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.583870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.583870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.583924 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.584213 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.584213 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.584248 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.584573 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.584664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":88,"threads_started":1}
I20260928 22:44:56.590241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6551
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 22:44:56.593406 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.586681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41096)
0928 22:44:56.586743 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.586753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.586795 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.586968 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.586969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.587085 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.587128 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.587130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.588001 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.588001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.588533 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.588535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.588997 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.588998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.589052 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.589404 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.589408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.590714 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.592270 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.592271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.592272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.592470 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.592659 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.592660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.592660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.592712 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.593121 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.593122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.593168 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.593224 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.593320 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:44:56.593667 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.586726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.98.60:42803)
0928 22:44:56.586859 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.586861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.586867 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:44:56.586898 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.586898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.586907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.586954 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.587191 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.587939 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.588602 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.588929 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.588999 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.590799 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.590800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.590801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.590807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.592186 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.592539 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.592540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.592540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.592591 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.592907 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.592907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.592908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.593034 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.593094 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.593510 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.593574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 22:44:56.597038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46165
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45071
--webserver_port=41269
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:44:56.597103 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42803
}
attrs {
promote: true
}
}
dest_uuid: "47571772bac54152978e413c10de5fe3"
cas_config_opid_index: 5
I20260928 22:44:56.597317 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } }
I20260928 22:44:56.597769 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.598711 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42803
I20260928 22:44:56.598793 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.598851 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:56.599469 7182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:56.600401 7182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:44:56.600582 7140 master.cc:537] Master@127.0.98.60:42803 shutting down...
I20260928 22:44:56.600766 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:44:56.601817 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 -> Peer aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60:42803): Couldn't send request to peer aecc4c8c3bcf4d55b97bc5190a6f62d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:44:56.603837 7140 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:56.603926 7140 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:56.603967 7140 tablet_replica.cc:330] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: stopping tablet replica
I20260928 22:44:56.605026 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.599060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43732)
0928 22:44:56.599114 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.599124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.599176 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.601569 (+ 2393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.601571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.601573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:56.601630 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.601633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.602530 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.602531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.603726 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.603729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.604757 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.604758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.604828 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.604855 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.604909 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.604953 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:44:56.605448 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.599262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.61:34473)
0928 22:44:56.601368 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.601371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.601381 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:44:56.601408 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.601409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.601433 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.601500 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:56.601706 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.602433 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.603828 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.604409 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.604479 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.604564 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.605332 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.605382 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078}
I20260928 22:44:56.605684 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 LEARNER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:44:56.606353 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 22:44:56.607102 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.608385 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47571772bac54152978e413c10de5fe3. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.608472 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: LEARNER
I20260928 22:44:56.608505 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: true } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.608547 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: LEARNER
I20260928 22:44:56.608832 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: attempting to promote NON_VOTER 20f52be26196463c9f6c89eb1cf7bb5d to VOTER
I20260928 22:44:56.609025 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } }
I20260928 22:44:56.609481 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 LEARNER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 22:44:56.609665 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 22:44:56.609732 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.609351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41112)
0928 22:44:56.609665 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.609704 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260928 22:44:56.610054 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.609937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41120)
0928 22:44:56.610008 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.610030 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 22:44:56.610118 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 -> Peer aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60:42803): Couldn't send request to peer aecc4c8c3bcf4d55b97bc5190a6f62d7. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:44:56.610414 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.610564 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.611224 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.611289 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:56.611331 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.611366 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:56.611472 7125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.611552 7125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:44:56.617537 7140 master.cc:552] Master@127.0.98.60:42803 shutdown complete.
I20260928 22:44:56.617589 7140 master_runner.cc:305] Clearing existing system tablet
I20260928 22:44:56.617704 7140 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.617758 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.618433 7140 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:56.618928 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.619032 7140 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.619074 7140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.619093 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "ca1af97e8b7bec48b0d44d9aba384ad2"
server_key_iv: "df6b6b031d7ab6b7f77a9ed091d780e3"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.619779 7140 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:56.621239 7140 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:44:56.621308 7140 log.cc:1196] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 22:44:56.621487 7140 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Deleting consensus metadata
I20260928 22:44:56.621594 7140 master_runner.cc:315] Copying system tablet from 127.0.98.62:45071
I20260928 22:44:56.622216 7140 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45071
I20260928 22:44:56.629114 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.622659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.98.62:45071)
0928 22:44:56.622701 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.622703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.622712 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:56.622747 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.622747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.622756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.622812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.623135 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.623896 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.624990 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.625284 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.625354 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.626388 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.626390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.626390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.626398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.627974 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.628348 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.628349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.628350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.628410 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.628585 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.628585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.628586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.628750 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.628812 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.628946 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.629027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:44:56.629122 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.622488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36134)
0928 22:44:56.622612 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.622624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.622688 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.622818 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.622819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.623011 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.623060 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.623062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.623984 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.623985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.624904 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.624908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.625365 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.625366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.625424 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.625900 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.625904 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.626297 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.628058 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.628059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.628060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.628269 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.628466 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.628467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.628467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.628530 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.628801 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.628802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.628849 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.628918 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.629019 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 22:44:56.629740 6509 tablet_copy_service.cc:140] P 47571772bac54152978e413c10de5fe3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aecc4c8c3bcf4d55b97bc5190a6f62d7 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36134)
I20260928 22:44:56.629792 6509 tablet_copy_service.cc:161] P 47571772bac54152978e413c10de5fe3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aecc4c8c3bcf4d55b97bc5190a6f62d7 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36134: session id = aecc4c8c3bcf4d55b97bc5190a6f62d7-00000000000000000000000000000000
I20260928 22:44:56.630342 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:44:56.630869 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:44:56.632102 7140 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: tablet copy: Starting download of 0 data blocks...
I20260928 22:44:56.632249 7140 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: tablet copy: Starting download of 1 WAL segments...
I20260928 22:44:56.633013 7140 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:44:56.633831 6509 tablet_copy_service.cc:342] P 47571772bac54152978e413c10de5fe3: Request end of tablet copy session aecc4c8c3bcf4d55b97bc5190a6f62d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36134
I20260928 22:44:56.633877 6509 tablet_copy_service.cc:433] P 47571772bac54152978e413c10de5fe3: ending tablet copy session aecc4c8c3bcf4d55b97bc5190a6f62d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aecc4c8c3bcf4d55b97bc5190a6f62d7
I20260928 22:44:56.636538 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.637701 7248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.637773 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.637928 7140 server_base.cc:1034] running on GCE node
W20260928 22:44:56.649596 7246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.649840 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.650008 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.651137 7140 hybrid_clock.cc:648] HybridClock initialized: now 1790635496651118 us; error 37 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.652614 7140 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 22:44:56.652947 7140 webserver.cc:460] Webserver started at http://127.0.98.60:33177/ using document root <none> and password file <none>
I20260928 22:44:56.653066 7140 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.653101 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.653828 7140 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:44:56.654374 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.654531 7140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.654584 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "ca1af97e8b7bec48b0d44d9aba384ad2"
server_key_iv: "df6b6b031d7ab6b7f77a9ed091d780e3"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.654635 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.669106 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.670951 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.674028 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42803
I20260928 22:44:56.674067 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42803 every 8 connection(s)
I20260928 22:44:56.674348 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
I20260928 22:44:56.677488 7308 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:44:56.678031 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Bootstrap starting.
I20260928 22:44:56.680900 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:44:56.681103 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Bootstrap complete.
I20260928 22:44:56.681289 7308 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } }
I20260928 22:44:56.681375 7308 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aecc4c8c3bcf4d55b97bc5190a6f62d7, State: Initialized, Role: LEARNER
I20260928 22:44:56.681419 7308 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } }
I20260928 22:44:56.681577 7308 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:44:56.683048 7322 catalog_manager.cc:1259] Loaded cluster ID: d9846d636e0b498bba87cf6c1b393722
I20260928 22:44:56.683284 7322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: loading cluster ID for follower catalog manager: success
I20260928 22:44:56.683669 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:56.683774 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:44:56.684676 7322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: acquiring CA information for follower catalog manager: success
I20260928 22:44:56.685026 7322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 22:44:56.691880 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.692070 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.692109 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.693276 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.693315 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.693351 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:44:56.694778 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41269
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:56.694995 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.695200 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.700796 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.700778 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.700795 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.701128 7234 server_base.cc:1034] running on GCE node
I20260928 22:44:56.701339 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.701609 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.702759 7234 hybrid_clock.cc:648] HybridClock initialized: now 1790635496702746 us; error 31 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.705082 7234 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 22:44:56.705479 7234 webserver.cc:460] Webserver started at http://127.0.98.1:41269/ using document root <none> and password file <none>
I20260928 22:44:56.705694 7234 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.705757 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.707022 7234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:56.707509 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.707660 7234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.707738 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "dad25693f8c04a6997fa58bf2991aaab"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "5cd7b2ab6d89c5cf1c57104a754bf533"
server_key_iv: "2a9d5fa29dc86c02637db9c16e80d5fb"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.708052 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.719002 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.721120 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.721302 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.721508 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:56.721794 7234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:56.721836 7234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.721869 7234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:56.721894 7234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.727488 7234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46165
I20260928 22:44:56.727550 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46165 every 8 connection(s)
I20260928 22:44:56.727857 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:44:56.732066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7234
I20260928 22:44:56.732220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6685
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 22:44:56.738095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43117
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45071
--webserver_port=46877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:44:56.739799 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46079 (local address 127.0.98.62:45071)
0928 22:44:56.728577 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.728579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.728860 (+ 281us) server_negotiation.cc:407] Connection header received
0928 22:44:56.729452 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.729454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.729466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.729513 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.730231 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.730997 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732226 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.732466 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732499 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.736905 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.736907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.736907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.736915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.738305 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.738766 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.738768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.738768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.738823 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739062 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.739063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.739063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.739195 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.739250 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.739624 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.739706 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:44:56.740022 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.98.1:46787)
0928 22:44:56.728734 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.728773 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.729285 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.729690 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.729690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.729936 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.730143 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.730149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.730993 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.730994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.732126 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.732127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.734751 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.734752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.734815 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.735172 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.735177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.736800 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.738584 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.738585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.738585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.738805 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.739006 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.739007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.739060 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.739330 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.739331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.739478 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.739587 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.739770 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":176,"thread_start_us":64,"threads_started":1}
I20260928 22:44:56.740144 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:46079)
0928 22:44:56.728701 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.728772 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.729276 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.729534 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.729539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.729941 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.730143 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.730148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.731069 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.731070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.732117 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.732119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.733442 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.733443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.733516 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.733881 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.733886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.736798 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.738389 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.738397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.738399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.738687 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.738883 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.738883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.738884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.739001 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.739297 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.739304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.739478 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.739918 (+ 440us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.740022 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":70,"thread_start_us":76,"threads_started":1}
I20260928 22:44:56.740337 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46787 (local address 127.0.98.60:42803)
0928 22:44:56.728827 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.728829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.728840 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:56.729570 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.729572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.729584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.729652 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.730255 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.730895 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732260 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.734678 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.734746 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.736905 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.736907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.736907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.736916 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.738504 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.738883 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.738884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.738884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.738941 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739120 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.739121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.739121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.739251 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.739306 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.740158 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.740246 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":90,"threads_started":1}
I20260928 22:44:56.740530 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:56.740617 7452 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.740625 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42803
I20260928 22:44:56.740669 7451 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.740828 7452 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:56.740828 7451 heartbeater.cc:511] Master 127.0.98.60:42803 requested a full tablet report, sending...
I20260928 22:44:56.741238 6489 ts_manager.cc:194] Re-registered known tserver with Master: dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
I20260928 22:44:56.741477 7272 ts_manager.cc:194] Registered new tserver with Master: dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
I20260928 22:44:56.741611 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49085 (local address 127.0.98.61:34473)
0928 22:44:56.728987 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.728989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.729013 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:44:56.729376 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.729377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.729390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.729449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.730255 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.730997 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732186 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.732469 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732510 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.736907 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.736909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.736909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.736916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.739357 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739686 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.739687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.739688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.739740 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.740972 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.740973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.740974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.741130 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.741165 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.741460 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.741532 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:44:56.741707 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46079
I20260928 22:44:56.742192 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.728856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.98.1:49085)
0928 22:44:56.728964 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.728973 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.729285 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.729841 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.729842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.730020 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.730143 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.730149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.731114 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.731115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.732029 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.732031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.732993 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.732994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.733113 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.733715 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.733724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.736793 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.739432 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.739433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.739615 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.739805 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.739805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.739806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.740875 (+ 1069us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.741248 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.741249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.741350 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.741879 (+ 529us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.741971 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":81,"thread_start_us":53,"threads_started":1}
I20260928 22:44:56.742478 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34473
I20260928 22:44:56.742524 7453 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.742583 7453 heartbeater.cc:511] Master 127.0.98.61:34473 requested a full tablet report, sending...
I20260928 22:44:56.742940 7086 ts_manager.cc:194] Registered new tserver with Master: dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
W20260928 22:44:56.818907 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.819113 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.819142 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.820266 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.820308 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.820344 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:44:56.821825 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46877
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:56.822054 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.822260 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.826963 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.826967 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.827010 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.827167 7459 server_base.cc:1034] running on GCE node
I20260928 22:44:56.827378 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.827592 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.828748 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790635496828735 us; error 29 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.831018 7459 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 22:44:56.831426 7459 webserver.cc:460] Webserver started at http://127.0.98.2:46877/ using document root <none> and password file <none>
I20260928 22:44:56.831594 7459 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.831641 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.832937 7459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:56.833474 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.833633 7459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.833710 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "59c6a5659bf64cef8715f0d82c50175a"
format_stamp: "Formatted at 2026-09-28 22:44:55 on dist-test-slave-pxr1"
server_key: "26b2acf003b6d83c418ab2683189916f"
server_key_iv: "177c0e4ecbb5719ebeafc0d5495a73b6"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.834046 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.852972 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.854908 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.855062 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.855275 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:56.855587 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:56.855628 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.855659 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:56.855682 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.861037 7459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43117
I20260928 22:44:56.861074 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43117 every 8 connection(s)
I20260928 22:44:56.861413 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:44:56.863152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7459
I20260928 22:44:56.863306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6819
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 22:44:56.869809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39077
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45071
--webserver_port=34743
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:44:56.872635 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36499 (local address 127.0.98.61:34473)
0928 22:44:56.862596 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.862598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.862645 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:44:56.863793 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.863794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.863807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.863835 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.864489 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.865288 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.867659 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.867950 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.868019 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.870049 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.870050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.870051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.870057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.871548 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871898 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.871899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.871899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.871953 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.872134 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.872134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.872135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.872267 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.872307 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.872485 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.872550 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:56.872640 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39815 (local address 127.0.98.60:42803)
0928 22:44:56.862462 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.862464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.862512 (+ 48us) server_negotiation.cc:407] Connection header received
0928 22:44:56.863021 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.863022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.863033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.863061 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.864548 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.865301 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.866777 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.866972 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.867025 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.869449 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.869451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.869451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.869458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.871201 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871653 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.871654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.871654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.871710 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871967 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.871968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.871968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.872104 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.872145 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.872450 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.872536 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:44:56.873006 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.98.2:36499)
0928 22:44:56.862591 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.862600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.862916 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.863911 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.863912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.864198 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.864401 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.864406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.866796 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.866796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.867578 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.867580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.868028 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.868029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.868097 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.868517 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.868522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.869958 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.871633 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.871635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.871829 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.872017 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.872018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.872018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.872071 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.872351 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.872352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.872386 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.872772 (+ 386us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.872867 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":62,"threads_started":1}
I20260928 22:44:56.873503 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34473
I20260928 22:44:56.873613 7590 heartbeater.cc:464] Registering TS with master...
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:44:56.873803 7590 heartbeater.cc:511] Master 127.0.98.61:34473 requested a full tablet report, sending...
I20260928 22:44:56.874249 7086 ts_manager.cc:194] Registered new tserver with Master: 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
I20260928 22:44:56.876415 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58761 (local address 127.0.98.62:45071)
0928 22:44:56.862201 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.862203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.862357 (+ 154us) server_negotiation.cc:407] Connection header received
0928 22:44:56.863035 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.863036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.863050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.863104 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.869209 (+ 6105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.870190 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.871721 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.872015 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.872052 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.874163 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.874165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.874165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.874172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.875452 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.875734 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.875735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.875735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.875789 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.875945 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.875945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.875946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.876053 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.876106 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.876254 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.876326 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:44:56.876423 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:58761)
0928 22:44:56.862228 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.862264 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.862908 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.863179 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.863184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.869051 (+ 5867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.869109 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.869112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.870648 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.870648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.871633 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.871635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.872120 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.872121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.872184 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.872829 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.872834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.874093 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.875524 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.875524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.875525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.875674 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.875839 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.875840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.875840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.875895 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.876104 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.876105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.876160 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.876217 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.876312 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260928 22:44:56.876782 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:56.876833 7589 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.876893 7589 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:56.877115 6489 ts_manager.cc:194] Re-registered known tserver with Master: 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
I20260928 22:44:56.877467 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58761
I20260928 22:44:56.880894 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.862347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.98.2:39815)
0928 22:44:56.862451 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.862461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.862912 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.864140 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.864141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.864335 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.864427 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.864429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.865404 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.865405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.866697 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.866700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.867048 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.867048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.867165 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.867635 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.867645 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.869345 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.871288 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.871297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.871571 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.871774 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.871774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.871775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.871895 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.872186 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.872190 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.872306 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.872415 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.872576 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":9,"thread_start_us":62,"threads_started":1}
I20260928 22:44:56.881294 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42803
I20260928 22:44:56.881366 7588 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.881444 7588 heartbeater.cc:511] Master 127.0.98.60:42803 requested a full tablet report, sending...
I20260928 22:44:56.881753 7272 ts_manager.cc:194] Registered new tserver with Master: 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
W20260928 22:44:56.951452 7595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:44:56.951659 7595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:44:56.951683 7595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:44:56.952852 7595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:44:56.952889 7595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:44:56.952915 7595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:44:56.954365 7595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34743
--tserver_master_addrs=127.0.98.62:45071,127.0.98.61:34473,127.0.98.60:42803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:44:56.954559 7595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:44:56.954766 7595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:44:56.959440 7601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.959444 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:44:56.959451 7603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:44:56.959723 7595 server_base.cc:1034] running on GCE node
I20260928 22:44:56.959919 7595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:44:56.960150 7595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:44:56.961282 7595 hybrid_clock.cc:648] HybridClock initialized: now 1790635496961265 us; error 27 us; skew 500 ppm
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:44:56.963660 7595 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 22:44:56.964064 7595 webserver.cc:460] Webserver started at http://127.0.98.3:34743/ using document root <none> and password file <none>
I20260928 22:44:56.964267 7595 fs_manager.cc:359] Metadata directory not provided
I20260928 22:44:56.964318 7595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:44:56.965703 7595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:56.966220 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.966377 7595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:44:56.966460 7595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "ee24594ebeba400aa5a000075698796b"
format_stamp: "Formatted at 2026-09-28 22:44:56 on dist-test-slave-pxr1"
server_key: "3c5f77a1fee93d73f860805f96f1b1b9"
server_key_iv: "5ea119007dadcf5c15d946c37422e233"
server_key_version: "encryptionkey@0"
I20260928 22:44:56.966784 7595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:44:56.974498 7595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:44:56.976279 7595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:44:56.976387 7595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:44:56.976570 7595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:44:56.976853 7595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:44:56.976887 7595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.976909 7595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:44:56.976925 7595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:56.982374 7595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39077
I20260928 22:44:56.982461 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39077 every 8 connection(s)
I20260928 22:44:56.982765 7595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:44:56.985419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7595
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 22:44:56 dist-test-slave-pxr1 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 22:44:56.995626 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.98.3:48793)
0928 22:44:56.983892 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.983901 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.984847 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.985071 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.985075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.985639 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.985892 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.985898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986859 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.986860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.987865 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.987868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988375 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.988669 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.989455 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.989460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.992313 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.994024 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.994035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.994331 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.994532 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.994533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.994663 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.994940 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.994947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.995113 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.995232 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.995393 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":14,"thread_start_us":49,"threads_started":1}
I20260928 22:44:56.995949 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.98.3:35371)
0928 22:44:56.983667 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.983714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.984844 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.985082 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.985083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.985571 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.985892 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.985898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986859 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.986860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.987863 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.987866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988364 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.988511 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.989381 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.989385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.992893 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.994713 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.994714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.994897 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.995089 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.995090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.995090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.995147 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.995415 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.995415 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.995458 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.995519 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.995634 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":141,"thread_start_us":81,"threads_started":1}
I20260928 22:44:56.995973 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35371 (local address 127.0.98.61:34473)
0928 22:44:56.983537 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.983539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.983807 (+ 268us) server_negotiation.cc:407] Connection header received
0928 22:44:56.984968 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.984969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.984983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.985013 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.986003 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986776 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.987960 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988260 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988308 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.992963 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.992964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.992964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.992971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.994658 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994971 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.994972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.994972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.995024 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.995209 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.995210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.995210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.995332 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.995390 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.995816 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.995894 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:44:56.996209 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48793 (local address 127.0.98.60:42803)
0928 22:44:56.983873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.983875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.983939 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:44:56.984937 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.984938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.984951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.984991 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.986002 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986767 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.987965 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988252 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988299 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.992392 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.992394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.992394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.992401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.993936 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994410 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.994412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.994412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.994467 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.994730 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.994731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.994731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.994858 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.994913 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.996026 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.996097 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:56.996533 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42803
I20260928 22:44:56.996634 7724 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.996861 7724 heartbeater.cc:511] Master 127.0.98.60:42803 requested a full tablet report, sending...
I20260928 22:44:56.996979 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34473
I20260928 22:44:56.997033 7726 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.997112 7726 heartbeater.cc:511] Master 127.0.98.61:34473 requested a full tablet report, sending...
I20260928 22:44:56.997280 7272 ts_manager.cc:194] Registered new tserver with Master: ee24594ebeba400aa5a000075698796b (127.0.98.3:39077)
I20260928 22:44:56.997396 7086 ts_manager.cc:194] Registered new tserver with Master: ee24594ebeba400aa5a000075698796b (127.0.98.3:39077)
I20260928 22:44:56.997491 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36603 (local address 127.0.98.62:45071)
0928 22:44:56.983548 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.983550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.983807 (+ 257us) server_negotiation.cc:407] Connection header received
0928 22:44:56.984968 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.984969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.984982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.985013 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.986008 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986621 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.987964 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988255 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988295 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.993838 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.993839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.993840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.993846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.995450 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.996675 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.996676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.996676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.996751 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.996944 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.996945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.996945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.997064 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.997102 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.997301 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.997386 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:44:56.997798 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.983510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:36603)
0928 22:44:56.983667 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:56.983714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:56.984844 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:56.985077 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:56.985078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:56.985624 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:44:56.985892 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.985899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.986732 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.986732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.987863 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:56.987866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.988369 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.988369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:56.988567 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.989211 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:44:56.989221 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:44:56.993430 (+ 4209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:44:56.995522 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.995523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.995523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.996595 (+ 1072us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.996821 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:44:56.996822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:44:56.996822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:44:56.996876 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:44:56.997144 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:44:56.997145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:44:56.997183 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 22:44:56.997605 (+ 422us) client_negotiation.cc:240] Negotiation successful
0928 22:44:56.997700 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":9,"spinlock_wait_cycles":7936,"thread_start_us":92,"threads_started":1}
I20260928 22:44:56.997968 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.989212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.0.98.61:34473)
0928 22:44:56.990169 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.990171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.990300 (+ 129us) server_negotiation.cc:407] Connection header received
0928 22:44:56.990323 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.990324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.990336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.990384 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.991173 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.991945 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.992924 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.993198 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.993253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.994424 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.994425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.994426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.994432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.996330 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.996707 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.996708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.996709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.996788 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.997104 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.997105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.997105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.997261 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.997760 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.997801 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.997880 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":52,"threads_started":1}
I20260928 22:44:56.998171 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45071
I20260928 22:44:56.998210 7725 heartbeater.cc:464] Registering TS with master...
I20260928 22:44:56.998262 7725 heartbeater.cc:511] Master 127.0.98.62:45071 requested a full tablet report, sending...
I20260928 22:44:56.998554 6489 ts_manager.cc:194] Re-registered known tserver with Master: ee24594ebeba400aa5a000075698796b (127.0.98.3:39077)
I20260928 22:44:56.998837 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.987195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.0.98.62:45071)
0928 22:44:56.988000 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.988002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.988145 (+ 143us) server_negotiation.cc:407] Connection header received
0928 22:44:56.988165 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.988166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.988177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.988226 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.989028 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.989719 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.990683 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.990993 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.991050 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:56.992819 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:56.992821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:56.992821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:56.992827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:56.997754 (+ 4927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.998112 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.998113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.998113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.998168 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:56.998361 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:56.998362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:56.998362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:56.998483 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:56.998578 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:56.998674 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 22:44:56.998750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":78,"threads_started":1}
I20260928 22:44:56.998996 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36603
I20260928 22:44:57.000806 6489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:57.002884 6489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:57.003108 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:56.989339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.60:42803)
0928 22:44:56.990226 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:56.990228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:56.990323 (+ 95us) server_negotiation.cc:407] Connection header received
0928 22:44:56.990400 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:56.990401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:56.990410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:56.990441 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:56.991179 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.991831 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.998104 (+ 6273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:56.998397 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:56.998443 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.000591 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:57.000593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:57.000593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:57.000600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:57.002018 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.002330 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.002331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.002331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.002390 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.002544 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.002544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.002545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.002661 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:57.002879 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.002917 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.003022 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":50,"threads_started":1}
I20260928 22:44:57.009949 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43117 (local address 127.0.0.1:47130)
0928 22:44:57.006481 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.006491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.006532 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.006928 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.006929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.006930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.006957 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.006958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.007805 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.007806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.008812 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.008814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.009709 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.009709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.009775 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.009779 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.009832 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.009873 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260928 22:44:57.009960 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.0.98.2:43117)
0928 22:44:57.006760 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.006764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.006775 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:44:57.006811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.006814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.006828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.007028 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.007081 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.007719 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.008984 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.009636 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.009698 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.009780 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.009846 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.009887 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260928 22:44:57.010098 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39077 (local address 127.0.0.1:33614)
0928 22:44:57.006529 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.006536 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.006565 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.006993 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.006994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.006996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.007057 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.007058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.008236 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.008237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.009042 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.009044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.009887 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.009888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.009954 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.009959 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.009986 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.010025 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260928 22:44:57.010300 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:53490)
0928 22:44:57.006404 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.006415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.006460 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.006836 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.006837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.006839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.006889 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.006891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.008182 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.008183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.009248 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.009250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.010093 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.010094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.010153 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.010156 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.010202 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.010236 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 22:44:57.010336 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.0.98.1:46165)
0928 22:44:57.006657 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.006662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.006671 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:57.006712 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.006714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.006729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.006845 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.006959 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.008101 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.009325 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.009999 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.010085 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.010154 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.010216 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.010257 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":72,"threads_started":1}
I20260928 22:44:57.010466 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.006519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.98.3:39077)
0928 22:44:57.006677 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.006681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.006688 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:57.006724 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.006726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.006738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.007119 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.007167 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.008149 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.009124 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.009792 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.010262 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.010339 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.010361 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.010396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 22:44:57.011009 7658 tablet_service.cc:1467] Processing CreateTablet for tablet 111a72a11d5f4e4e829eacba433e138d (DEFAULT_TABLE table=test-table [id=8597dde458e74ca2834df72a8b437515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:57.011008 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 111a72a11d5f4e4e829eacba433e138d (DEFAULT_TABLE table=test-table [id=8597dde458e74ca2834df72a8b437515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:57.011344 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111a72a11d5f4e4e829eacba433e138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:57.011363 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111a72a11d5f4e4e829eacba433e138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:57.011327 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 111a72a11d5f4e4e829eacba433e138d (DEFAULT_TABLE table=test-table [id=8597dde458e74ca2834df72a8b437515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:57.011602 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111a72a11d5f4e4e829eacba433e138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:57.014302 7750 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Bootstrap starting.
I20260928 22:44:57.014953 7751 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Bootstrap starting.
I20260928 22:44:57.014896 7750 tablet_bootstrap.cc:654] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:57.015388 7750 log.cc:824] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:57.015520 7751 tablet_bootstrap.cc:654] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:57.015826 7751 log.cc:824] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:57.015928 7752 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Bootstrap starting.
I20260928 22:44:57.016098 7750 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: No bootstrap required, opened a new log
I20260928 22:44:57.016157 7750 ts_tablet_manager.cc:1397] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:57.016490 7751 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: No bootstrap required, opened a new log
I20260928 22:44:57.016490 7752 tablet_bootstrap.cc:654] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:57.016556 7751 ts_tablet_manager.cc:1397] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:44:57.016799 7752 log.cc:824] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Log is configured to *not* fsync() on all Append() calls
I20260928 22:44:57.017524 7752 tablet_bootstrap.cc:492] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: No bootstrap required, opened a new log
I20260928 22:44:57.017589 7752 ts_tablet_manager.cc:1397] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:44:57.018093 7750 raft_consensus.cc:348] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.018219 7750 raft_consensus.cc:374] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:57.018249 7750 raft_consensus.cc:729] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee24594ebeba400aa5a000075698796b, State: Initialized, Role: FOLLOWER
I20260928 22:44:57.018354 7750 consensus_queue.cc:260] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.018596 7750 ts_tablet_manager.cc:1428] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:44:57.018682 7725 heartbeater.cc:503] Master 127.0.98.62:45071 was elected leader, sending a full tablet report...
I20260928 22:44:57.019382 7751 raft_consensus.cc:348] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.019536 7751 raft_consensus.cc:374] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:57.019575 7751 raft_consensus.cc:729] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c6a5659bf64cef8715f0d82c50175a, State: Initialized, Role: FOLLOWER
I20260928 22:44:57.019681 7751 consensus_queue.cc:260] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.019942 7751 ts_tablet_manager.cc:1428] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:44:57.019959 7589 heartbeater.cc:503] Master 127.0.98.62:45071 was elected leader, sending a full tablet report...
I20260928 22:44:57.020440 7752 raft_consensus.cc:348] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.020591 7752 raft_consensus.cc:374] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:57.020634 7752 raft_consensus.cc:729] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad25693f8c04a6997fa58bf2991aaab, State: Initialized, Role: FOLLOWER
I20260928 22:44:57.020805 7752 consensus_queue.cc:260] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.021059 7752 ts_tablet_manager.cc:1428] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:44:57.021170 7452 heartbeater.cc:503] Master 127.0.98.62:45071 was elected leader, sending a full tablet report...
I20260928 22:44:57.024206 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.021638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.0.98.60:42803)
0928 22:44:57.021675 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.021676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.021684 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:57.021705 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.021706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.021725 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.021782 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.021931 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.022514 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.023376 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.023920 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.023977 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.024046 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.024102 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.024139 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:44:57.024209 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.021536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41132)
0928 22:44:57.021579 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.021589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.021624 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.021826 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.021827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.021829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.021872 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.021873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.022587 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.022588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.023298 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.023300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.023978 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.023979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.024041 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.024045 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.024096 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.024140 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 22:44:57.024525 7282 raft_consensus.cc:1203] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 22:44:57.025290 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47571772bac54152978e413c10de5fe3. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:57.025382 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:44:57.109401 7756 raft_consensus.cc:482] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:57.109520 7756 raft_consensus.cc:504] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.109823 7756 leader_election.cc:288] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165), 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
W20260928 22:44:57.112890 7591 tablet.cc:2369] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:57.113586 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.110116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58563 (local address 127.0.98.1:46165)
0928 22:44:57.110163 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.110165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.110173 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:57.110200 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.110200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.110208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.110256 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.110429 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.111180 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.112650 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.113246 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.113317 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.113327 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.113482 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.113518 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:44:57.113586 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.110092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44735 (local address 127.0.98.2:43117)
0928 22:44:57.110133 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.110135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.110213 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:44:57.110240 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.110241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.110252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.110293 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:57.110432 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.111173 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.112661 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.113246 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.113317 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.113327 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.113483 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.113518 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:57.113610 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.109998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.98.3:58563)
0928 22:44:57.110045 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.110057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.110110 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.110278 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.110279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.110281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.110363 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.110364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.111274 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.111275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.112557 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.112560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.113323 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.113324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.113415 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.113417 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.113472 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.113516 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":65}
I20260928 22:44:57.113673 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.109999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43117 (local address 127.0.98.3:44735)
0928 22:44:57.110133 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:57.110141 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:57.110176 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:57.110329 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:57.110330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:57.110331 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:57.110376 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.110377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.111274 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.111275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.112557 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:57.112560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.113323 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.113324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:57.113408 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.113412 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:57.113472 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:44:57.113516 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":121}
I20260928 22:44:57.113958 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111a72a11d5f4e4e829eacba433e138d" candidate_uuid: "ee24594ebeba400aa5a000075698796b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c6a5659bf64cef8715f0d82c50175a" is_pre_election: true
I20260928 22:44:57.114112 7542 raft_consensus.cc:2454] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee24594ebeba400aa5a000075698796b in term 0.
I20260928 22:44:57.114147 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111a72a11d5f4e4e829eacba433e138d" candidate_uuid: "ee24594ebeba400aa5a000075698796b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad25693f8c04a6997fa58bf2991aaab" is_pre_election: true
I20260928 22:44:57.114300 7405 raft_consensus.cc:2454] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee24594ebeba400aa5a000075698796b in term 0.
I20260928 22:44:57.114331 7613 leader_election.cc:302] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c6a5659bf64cef8715f0d82c50175a, ee24594ebeba400aa5a000075698796b; no voters:
I20260928 22:44:57.114460 7756 raft_consensus.cc:2781] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:57.114524 7756 raft_consensus.cc:482] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:57.114552 7756 raft_consensus.cc:3037] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:57.115235 7756 raft_consensus.cc:504] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.115356 7756 leader_election.cc:288] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [CANDIDATE]: Term 1 election: Requested vote from peers dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165), 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117)
I20260928 22:44:57.115477 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111a72a11d5f4e4e829eacba433e138d" candidate_uuid: "ee24594ebeba400aa5a000075698796b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad25693f8c04a6997fa58bf2991aaab"
I20260928 22:44:57.115490 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111a72a11d5f4e4e829eacba433e138d" candidate_uuid: "ee24594ebeba400aa5a000075698796b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c6a5659bf64cef8715f0d82c50175a"
I20260928 22:44:57.115553 7542 raft_consensus.cc:3037] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:57.115553 7405 raft_consensus.cc:3037] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:57.116396 7542 raft_consensus.cc:2454] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee24594ebeba400aa5a000075698796b in term 1.
I20260928 22:44:57.116396 7405 raft_consensus.cc:2454] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee24594ebeba400aa5a000075698796b in term 1.
I20260928 22:44:57.116582 7613 leader_election.cc:302] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c6a5659bf64cef8715f0d82c50175a, ee24594ebeba400aa5a000075698796b; no voters:
I20260928 22:44:57.116660 7756 raft_consensus.cc:2781] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:57.116835 7756 raft_consensus.cc:686] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 1 LEADER]: Becoming Leader. State: Replica: ee24594ebeba400aa5a000075698796b, State: Running, Role: LEADER
I20260928 22:44:57.116927 7756 consensus_queue.cc:237] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } }
I20260928 22:44:57.117729 6488 catalog_manager.cc:5495] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee24594ebeba400aa5a000075698796b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ee24594ebeba400aa5a000075698796b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } health_report { overall_health: HEALTHY } } }
I20260928 22:44:57.118247 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: true } } }
I20260928 22:44:57.118332 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:57.119127 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: attempting to promote NON_VOTER aecc4c8c3bcf4d55b97bc5190a6f62d7 to VOTER
I20260928 22:44:57.119261 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } }
I20260928 22:44:57.119668 7282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 LEARNER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 22:44:57.119700 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Refusing update from remote peer 47571772bac54152978e413c10de5fe3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 22:44:57.119840 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 22:44:57.119932 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 22:44:57.120523 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.120657 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.120684 7282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121302 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121376 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:57.121428 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121474 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:57.121510 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121554 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:57.121608 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121654 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47571772bac54152978e413c10de5fe3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:57.121711 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:44:57.121727 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:44:57.130498 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.128310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.0.98.3:39077)
0928 22:44:57.128368 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.128371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.128378 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:57.128411 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.128412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.128433 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.128490 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:57.128623 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.129123 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.129822 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.130019 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.130071 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.130211 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:57.130317 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:57.130386 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.130405 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.130426 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:57.132107 7405 raft_consensus.cc:1261] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Refusing update from remote peer ee24594ebeba400aa5a000075698796b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:57.132419 7542 raft_consensus.cc:1261] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Refusing update from remote peer ee24594ebeba400aa5a000075698796b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:57.132413 7762 consensus_queue.cc:1035] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:57.132659 7762 consensus_queue.cc:1035] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:57.135303 7768 mvcc.cc:205] Tried to move back new op lower bound from 7334442996250935296 to 7334442996191629312. Current Snapshot: MvccSnapshot[applied={T|T < 7334442996250935296}]
I20260928 22:44:57.135602 7769 mvcc.cc:205] Tried to move back new op lower bound from 7334442996250935296 to 7334442996191629312. Current Snapshot: MvccSnapshot[applied={T|T < 7334442996250935296}]
I20260928 22:44:57.136287 7770 mvcc.cc:205] Tried to move back new op lower bound from 7334442996250935296 to 7334442996191629312. Current Snapshot: MvccSnapshot[applied={T|T < 7334442996250935296}]
I20260928 22:44:57.138072 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:57.138149 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:57.139266 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47571772bac54152978e413c10de5fe3: Sending DeleteTablet for 3 replicas of tablet 111a72a11d5f4e4e829eacba433e138d
I20260928 22:44:57.139581 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet 111a72a11d5f4e4e829eacba433e138d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47130
I20260928 22:44:57.139667 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet 111a72a11d5f4e4e829eacba433e138d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33614
I20260928 22:44:57.139842 7778 tablet_replica.cc:330] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: stopping tablet replica
I20260928 22:44:57.139947 7778 raft_consensus.cc:2229] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:57.140040 7778 raft_consensus.cc:2258] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:57.140439 7778 ts_tablet_manager.cc:1905] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:57.140478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6457
I20260928 22:44:57.140864 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 111a72a11d5f4e4e829eacba433e138d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53490
I20260928 22:44:57.142235 7778 ts_tablet_manager.cc:1918] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:57.142304 7778 log.cc:1196] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/111a72a11d5f4e4e829eacba433e138d
I20260928 22:44:57.142545 7778 ts_tablet_manager.cc:1939] T 111a72a11d5f4e4e829eacba433e138d P 59c6a5659bf64cef8715f0d82c50175a: Deleting consensus metadata
I20260928 22:44:57.144028 7779 tablet_replica.cc:330] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: stopping tablet replica
I20260928 22:44:57.144125 7779 raft_consensus.cc:2229] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:57.144212 7779 raft_consensus.cc:2258] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:57.145416 7779 ts_tablet_manager.cc:1905] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:57.146919 7779 ts_tablet_manager.cc:1918] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:57.146975 7779 log.cc:1196] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/111a72a11d5f4e4e829eacba433e138d
I20260928 22:44:57.147210 7779 ts_tablet_manager.cc:1939] T 111a72a11d5f4e4e829eacba433e138d P ee24594ebeba400aa5a000075698796b: Deleting consensus metadata
W20260928 22:44:57.147341 7478 connection.cc:535] server connection from 127.0.0.1:47130 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:44:57.147584 7613 connection.cc:441] server connection from 127.0.0.1:33614 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33614 (request call id 1) could send its response
I20260928 22:44:57.150106 7781 tablet_replica.cc:330] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: stopping tablet replica
I20260928 22:44:57.150229 7781 raft_consensus.cc:2229] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:57.150389 7781 raft_consensus.cc:2258] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:57.151237 7781 ts_tablet_manager.cc:1905] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:57.152756 7781 ts_tablet_manager.cc:1918] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:57.152838 7781 log.cc:1196] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/111a72a11d5f4e4e829eacba433e138d
I20260928 22:44:57.153086 7781 ts_tablet_manager.cc:1939] T 111a72a11d5f4e4e829eacba433e138d P dad25693f8c04a6997fa58bf2991aaab: Deleting consensus metadata
W20260928 22:44:57.153441 7341 connection.cc:441] server connection from 127.0.0.1:53490 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53490 (request call id 1) could send its response
I20260928 22:44:57.154829 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.140316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.0.98.60:42803)
0928 22:44:57.140355 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.140357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.141381 (+ 1024us) server_negotiation.cc:407] Connection header received
0928 22:44:57.141471 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.141472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.141481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.141507 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:57.142223 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.142932 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.144339 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.144629 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.144670 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.149026 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:57.149028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:57.149028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:57.149036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:57.150682 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.153816 (+ 3134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.153817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.153818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.153875 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.154066 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.154067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.154067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.154190 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:57.154275 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.154639 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.154709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:44:57.156318 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.149358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.0.98.60:42803)
0928 22:44:57.149396 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.149398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.149435 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:44:57.149522 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.149523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.149532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.149570 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:57.150150 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.150921 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.151991 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.152242 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.152285 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.153628 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:57.153629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:57.153629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:57.153636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:57.155037 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.155537 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.155538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.155539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.155607 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.155788 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.155789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.155789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.155910 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:57.155999 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.156160 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.156237 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:44:57.158041 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:57.149001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.98.61:34473)
0928 22:44:57.149042 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:57.149044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:57.149179 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:44:57.149274 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:57.149275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:57.149552 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:57.149585 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:44:57.150558 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.151331 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.152427 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:57.153122 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:57.153167 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:57.155489 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:44:57.155490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:44:57.155491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:44:57.155515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:44:57.157095 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.157434 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.157435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.157435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.157482 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:44:57.157648 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:44:57.157649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:44:57.157649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:44:57.157746 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:44:57.157808 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:57.157923 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:44:57.157982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:44:58.137526 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.137266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:53411)
0928 22:44:58.137389 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.137471 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 22:44:58.137526 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.137267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:54411)
0928 22:44:58.137401 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.137478 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260928 22:44:58.137763 7589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
W20260928 22:44:58.137766 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.138299 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.138111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:46913)
0928 22:44:58.138206 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.138265 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
W20260928 22:44:58.138486 7725 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.640012 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47571772bac54152978e413c10de5fe3)
I20260928 22:44:58.640136 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } }
I20260928 22:44:58.640352 7792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47571772bac54152978e413c10de5fe3)
I20260928 22:44:58.640417 7791 leader_election.cc:288] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071), 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61:34473)
I20260928 22:44:58.640728 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.640523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36636)
0928 22:44:58.640641 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.640679 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260928 22:44:58.640708 7792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } }
W20260928 22:44:58.640837 7256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.641199 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.641118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36650)
0928 22:44:58.641152 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.641174 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 22:44:58.641263 7256 leader_election.cc:334] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.641302 7792 leader_election.cc:288] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071), aecc4c8c3bcf4d55b97bc5190a6f62d7 (127.0.98.60:42803)
I20260928 22:44:58.641479 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.641062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36640)
0928 22:44:58.641158 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.641218 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
W20260928 22:44:58.641585 7070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.641940 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.641853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36662)
0928 22:44:58.641891 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.641915 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:44:58.642078 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.643584 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.640554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34473 (local address 127.0.0.1:43766)
0928 22:44:58.640792 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.640801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:58.640865 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:58.641064 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:58.641065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:58.641067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:58.641118 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.641120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.642008 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.642009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.642828 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.642830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.643360 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.643361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.643423 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.643427 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:58.643473 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:44:58.643516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":40,"threads_started":1}
I20260928 22:44:58.643577 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.640615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.61:34473)
0928 22:44:58.640741 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:58.640743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:58.640920 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:44:58.640949 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:58.640950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:58.640960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:58.641013 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:58.641190 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.641929 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.642913 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.643292 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.643352 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.643363 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:58.643482 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 22:44:58.643517 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260928 22:44:58.643782 7096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" is_pre_election: true
I20260928 22:44:58.643882 7096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecc4c8c3bcf4d55b97bc5190a6f62d7 in term 1.
I20260928 22:44:58.643940 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.641309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.0.98.60:42803)
0928 22:44:58.641453 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:58.641454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:58.641461 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:44:58.641481 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:58.641482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:58.641490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:58.641550 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:58.641687 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.642326 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.643236 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.643610 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.643669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.643681 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:58.643845 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 22:44:58.643887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260928 22:44:58.643960 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.641238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42803 (local address 127.0.0.1:41152)
0928 22:44:58.641384 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.641392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:58.641439 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:58.641589 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:58.641590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:58.641591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:58.641627 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.641629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.642394 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.642395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.643157 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.643159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.643716 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.643717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.643776 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.643780 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:58.643828 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:58.643865 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":42,"threads_started":1}
I20260928 22:44:58.644043 7259 leader_election.cc:302] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20f52be26196463c9f6c89eb1cf7bb5d, aecc4c8c3bcf4d55b97bc5190a6f62d7; no voters: 47571772bac54152978e413c10de5fe3
I20260928 22:44:58.644160 7791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 22:44:58.644209 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47571772bac54152978e413c10de5fe3)
I20260928 22:44:58.644255 7791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:44:58.644198 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" is_pre_election: true
I20260928 22:44:58.644954 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } }
I20260928 22:44:58.645071 7282 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20f52be26196463c9f6c89eb1cf7bb5d in current term 2: Already voted for candidate aecc4c8c3bcf4d55b97bc5190a6f62d7 in this term.
I20260928 22:44:58.645076 7791 leader_election.cc:288] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 election: Requested vote from peers 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071), 20f52be26196463c9f6c89eb1cf7bb5d (127.0.98.61:34473)
I20260928 22:44:58.645231 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.645152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36672)
0928 22:44:58.645182 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.645208 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 22:44:58.645241 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f52be26196463c9f6c89eb1cf7bb5d; no voters: 47571772bac54152978e413c10de5fe3, aecc4c8c3bcf4d55b97bc5190a6f62d7
I20260928 22:44:58.645222 7096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20f52be26196463c9f6c89eb1cf7bb5d"
I20260928 22:44:58.645298 7096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:44:58.645505 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.645399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36682)
0928 22:44:58.645434 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.645485 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 22:44:58.645573 7256 leader_election.cc:334] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
I20260928 22:44:58.645985 7096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecc4c8c3bcf4d55b97bc5190a6f62d7 in term 2.
I20260928 22:44:58.646050 7792 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:44:58.646147 7259 leader_election.cc:302] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20f52be26196463c9f6c89eb1cf7bb5d, aecc4c8c3bcf4d55b97bc5190a6f62d7; no voters: 47571772bac54152978e413c10de5fe3
I20260928 22:44:58.646234 7791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:44:58.646371 7791 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [term 2 LEADER]: Becoming Leader. State: Replica: aecc4c8c3bcf4d55b97bc5190a6f62d7, State: Running, Role: LEADER
I20260928 22:44:58.646430 7791 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } }
I20260928 22:44:58.646733 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecc4c8c3bcf4d55b97bc5190a6f62d7. Latest consensus state: current_term: 2 leader_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:58.646816 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:58.646984 7803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:44:58.647398 7803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:44:58.647471 7803 catalog_manager.cc:1259] Loaded cluster ID: d9846d636e0b498bba87cf6c1b393722
I20260928 22:44:58.647501 7803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:44:58.647660 7803 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:44:58.647742 7803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Loaded TSK: 0
I20260928 22:44:58.647845 7803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:44:58.691429 7271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:44:58.693184 7271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:44:58.694199 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.694064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36688)
0928 22:44:58.694132 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.694167 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 22:44:58.694195 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [term 2 FOLLOWER]: Refusing update from remote peer aecc4c8c3bcf4d55b97bc5190a6f62d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 22:44:58.694415 7801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 22:44:58.694537 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.694441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:36696)
0928 22:44:58.694486 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.694512 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:44:58.694653 7256 consensus_peers.cc:487] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 -> Peer 47571772bac54152978e413c10de5fe3 (127.0.98.62:45071): Couldn't send request to peer 47571772bac54152978e413c10de5fe3. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:44:58.695020 7792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecc4c8c3bcf4d55b97bc5190a6f62d7. Latest consensus state: current_term: 2 leader_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:58.695108 7792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:44:58.695492 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:58.695672 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:44:58.696118 7792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47571772bac54152978e413c10de5fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } peers { permanent_uuid: "20f52be26196463c9f6c89eb1cf7bb5d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34473 } attrs { promote: false } } peers { permanent_uuid: "aecc4c8c3bcf4d55b97bc5190a6f62d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42803 } attrs { promote: false } } }
I20260928 22:44:58.696209 7792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f52be26196463c9f6c89eb1cf7bb5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:44:58.696285 7805 mvcc.cc:205] Tried to move back new op lower bound from 7334443002649559040 to 7334443002456363008. Current Snapshot: MvccSnapshot[applied={T|T < 7334443002649559040 or (T in {7334443002649559040})}]
I20260928 22:44:58.700438 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:53496)
0928 22:44:58.697760 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.697812 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:58.697899 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:58.698027 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:58.698028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:58.698029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:58.698074 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.698076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698704 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.698705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.699484 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.699486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.700229 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.700230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.700283 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.700287 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:58.700344 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:44:58.700379 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":85,"thread_start_us":78,"threads_started":1}
I20260928 22:44:58.700469 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.0.98.1:46165)
0928 22:44:58.697643 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:58.697645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:58.697877 (+ 232us) server_negotiation.cc:407] Connection header received
0928 22:44:58.697964 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:58.697965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:58.697975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:58.698027 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:58.698121 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698613 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.699574 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.700131 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.700207 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.700221 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:58.700356 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:44:58.700395 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260928 22:44:58.700807 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.0.98.2:43117)
0928 22:44:58.697517 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:58.697519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:58.697527 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:58.697552 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:58.697553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:58.697563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:58.697675 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:58.697794 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698524 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.699884 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.700401 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.700477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.700490 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:58.700659 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:44:58.700700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260928 22:44:58.700816 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43117 (local address 127.0.0.1:47146)
0928 22:44:58.697232 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.697242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:58.697290 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:58.697677 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:58.697677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:58.697678 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:58.697713 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.697713 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698601 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.698602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.699799 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.699801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.700520 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.700521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.700593 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.700597 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:58.700652 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:44:58.700689 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 22:44:58.701233 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 082ff898b2354e688e801a50ac191152 (DEFAULT_TABLE table=test-table [id=ee61752682034ab9b35d29a6293b68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:58.701277 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 082ff898b2354e688e801a50ac191152 (DEFAULT_TABLE table=test-table [id=ee61752682034ab9b35d29a6293b68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:58.701342 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39077 (local address 127.0.0.1:33640)
0928 22:44:58.697294 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.697304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:58.697339 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:58.697653 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:58.697654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:58.697656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:58.697708 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.697710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698749 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.698750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.699809 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:58.699810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.701135 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.701136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:58.701195 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.701199 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:44:58.701247 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:58.701286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":33}
I20260928 22:44:58.701361 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082ff898b2354e688e801a50ac191152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:58.701354 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.697415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.0.98.3:39077)
0928 22:44:58.697522 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:58.697524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:58.697532 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:44:58.697555 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:58.697556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:58.697564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:58.697770 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:58.697807 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.698659 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.700530 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:58.701072 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:58.701133 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:58.701145 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:58.701252 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:44:58.701292 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260928 22:44:58.701395 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082ff898b2354e688e801a50ac191152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:58.702490 7813 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Bootstrap starting.
I20260928 22:44:58.702903 7813 tablet_bootstrap.cc:654] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:58.703024 7658 tablet_service.cc:1467] Processing CreateTablet for tablet 082ff898b2354e688e801a50ac191152 (DEFAULT_TABLE table=test-table [id=ee61752682034ab9b35d29a6293b68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:44:58.703141 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082ff898b2354e688e801a50ac191152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:44:58.703181 7814 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: Bootstrap starting.
I20260928 22:44:58.703490 7814 tablet_bootstrap.cc:654] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:58.703815 7813 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: No bootstrap required, opened a new log
I20260928 22:44:58.703866 7813 ts_tablet_manager.cc:1397] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:58.703974 7813 raft_consensus.cc:348] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.704028 7813 raft_consensus.cc:374] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:58.704039 7813 raft_consensus.cc:729] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c6a5659bf64cef8715f0d82c50175a, State: Initialized, Role: FOLLOWER
I20260928 22:44:58.704072 7813 consensus_queue.cc:260] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.704219 7813 ts_tablet_manager.cc:1428] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:58.704397 7818 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Bootstrap starting.
I20260928 22:44:58.704566 7814 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: No bootstrap required, opened a new log
I20260928 22:44:58.704618 7814 ts_tablet_manager.cc:1397] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:44:58.704785 7818 tablet_bootstrap.cc:654] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Neither blocks nor log segments found. Creating new log.
I20260928 22:44:58.704703 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.704368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:35067)
0928 22:44:58.704535 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.704582 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260928 22:44:58.704998 7588 heartbeater.cc:503] Master 127.0.98.60:42803 was elected leader, sending a full tablet report...
I20260928 22:44:58.705046 7814 raft_consensus.cc:348] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.705117 7814 raft_consensus.cc:374] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:58.705142 7814 raft_consensus.cc:729] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad25693f8c04a6997fa58bf2991aaab, State: Initialized, Role: FOLLOWER
I20260928 22:44:58.705200 7814 consensus_queue.cc:260] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.705240 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.705085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:50329)
0928 22:44:58.705190 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.705214 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260928 22:44:58.705330 7814 ts_tablet_manager.cc:1428] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:58.705879 7451 heartbeater.cc:503] Master 127.0.98.60:42803 was elected leader, sending a full tablet report...
I20260928 22:44:58.705896 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.705700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:59569)
0928 22:44:58.705823 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.705858 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 22:44:58.705983 7818 tablet_bootstrap.cc:492] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: No bootstrap required, opened a new log
I20260928 22:44:58.706023 7818 ts_tablet_manager.cc:1397] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:44:58.706128 7818 raft_consensus.cc:348] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.706197 7818 raft_consensus.cc:374] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:44:58.706223 7818 raft_consensus.cc:729] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee24594ebeba400aa5a000075698796b, State: Initialized, Role: FOLLOWER
I20260928 22:44:58.706259 7818 consensus_queue.cc:260] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:58.706383 7818 ts_tablet_manager.cc:1428] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:44:58.706410 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.706324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:35959)
0928 22:44:58.706358 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.706386 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 22:44:58.706810 7724 heartbeater.cc:503] Master 127.0.98.60:42803 was elected leader, sending a full tablet report...
I20260928 22:44:58.706866 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.706636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:42295)
0928 22:44:58.706800 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.706831 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260928 22:44:58.707212 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:58.707113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:40271)
0928 22:44:58.707160 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:58.707189 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:44:58.729589 7454 tablet.cc:2369] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:44:58.734331 7727 tablet.cc:2369] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:44:59.005151 7817 raft_consensus.cc:482] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:44:59.005256 7817 raft_consensus.cc:504] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:59.005578 7817 leader_election.cc:288] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee24594ebeba400aa5a000075698796b (127.0.98.3:39077), dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
I20260928 22:44:59.008425 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.005864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49257 (local address 127.0.98.3:39077)
0928 22:44:59.005917 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:59.005919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:59.005931 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:44:59.005971 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:59.005972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:59.005986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:59.006047 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:59.006212 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.006744 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.007752 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.008105 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.008163 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:59.008177 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:59.008324 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 22:44:59.008362 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:44:59.008432 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.005844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41757 (local address 127.0.98.1:46165)
0928 22:44:59.005911 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:59.005912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:59.006082 (+ 170us) server_negotiation.cc:407] Connection header received
0928 22:44:59.006113 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:59.006115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:59.006124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:59.006176 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:44:59.006307 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.006764 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.007739 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.008104 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.008163 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:59.008176 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:59.008316 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 22:44:59.008350 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:44:59.008425 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.005775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.98.2:41757)
0928 22:44:59.006004 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.006015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:59.006068 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:59.006191 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:59.006192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:59.006193 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:59.006256 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:59.006258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.006851 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.006851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:59.007642 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:59.007644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.008182 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.008183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:59.008248 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:59.008251 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:59.008293 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:44:59.008338 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":181}
I20260928 22:44:59.008435 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.005776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39077 (local address 127.0.98.2:49257)
0928 22:44:59.005840 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.005852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:59.005908 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:59.006069 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:44:59.006070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:44:59.006072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:44:59.006148 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:59.006151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.006826 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.006827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:59.007658 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:44:59.007660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.008180 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.008181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:44:59.008255 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:59.008258 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:44:59.008306 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:44:59.008354 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":43}
I20260928 22:44:59.008659 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082ff898b2354e688e801a50ac191152" candidate_uuid: "59c6a5659bf64cef8715f0d82c50175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad25693f8c04a6997fa58bf2991aaab" is_pre_election: true
I20260928 22:44:59.008769 7405 raft_consensus.cc:2454] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59c6a5659bf64cef8715f0d82c50175a in term 0.
I20260928 22:44:59.008756 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082ff898b2354e688e801a50ac191152" candidate_uuid: "59c6a5659bf64cef8715f0d82c50175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee24594ebeba400aa5a000075698796b" is_pre_election: true
I20260928 22:44:59.008898 7678 raft_consensus.cc:2454] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59c6a5659bf64cef8715f0d82c50175a in term 0.
I20260928 22:44:59.008921 7478 leader_election.cc:302] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c6a5659bf64cef8715f0d82c50175a, dad25693f8c04a6997fa58bf2991aaab; no voters:
I20260928 22:44:59.009084 7817 raft_consensus.cc:2781] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:44:59.009145 7817 raft_consensus.cc:482] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:44:59.009166 7817 raft_consensus.cc:3037] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:59.009862 7817 raft_consensus.cc:504] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:59.009994 7817 leader_election.cc:288] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [CANDIDATE]: Term 1 election: Requested vote from peers ee24594ebeba400aa5a000075698796b (127.0.98.3:39077), dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165)
I20260928 22:44:59.010197 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082ff898b2354e688e801a50ac191152" candidate_uuid: "59c6a5659bf64cef8715f0d82c50175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee24594ebeba400aa5a000075698796b"
I20260928 22:44:59.010185 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082ff898b2354e688e801a50ac191152" candidate_uuid: "59c6a5659bf64cef8715f0d82c50175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad25693f8c04a6997fa58bf2991aaab"
I20260928 22:44:59.010277 7405 raft_consensus.cc:3037] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:59.010285 7678 raft_consensus.cc:3037] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:44:59.011039 7405 raft_consensus.cc:2454] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59c6a5659bf64cef8715f0d82c50175a in term 1.
I20260928 22:44:59.011092 7678 raft_consensus.cc:2454] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59c6a5659bf64cef8715f0d82c50175a in term 1.
I20260928 22:44:59.011210 7478 leader_election.cc:302] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c6a5659bf64cef8715f0d82c50175a, dad25693f8c04a6997fa58bf2991aaab; no voters:
I20260928 22:44:59.011345 7817 raft_consensus.cc:2781] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:44:59.011509 7817 raft_consensus.cc:686] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 1 LEADER]: Becoming Leader. State: Replica: 59c6a5659bf64cef8715f0d82c50175a, State: Running, Role: LEADER
I20260928 22:44:59.011577 7817 consensus_queue.cc:237] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } }
I20260928 22:44:59.011819 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.011722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:33497)
0928 22:44:59.011758 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.011790 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 22:44:59.012286 7270 catalog_manager.cc:5495] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a reported cstate change: term changed from 0 to 1, leader changed from <none> to 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59c6a5659bf64cef8715f0d82c50175a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c6a5659bf64cef8715f0d82c50175a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 } health_report { overall_health: UNKNOWN } } }
I20260928 22:44:59.052626 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.050465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.98.2:43117)
0928 22:44:59.050520 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:44:59.050521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:44:59.050530 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:44:59.050588 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:44:59.050588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:44:59.050620 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:44:59.050682 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:44:59.050768 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.051250 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.051976 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:44:59.052168 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:44:59.052221 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:44:59.052320 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:44:59.052420 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:44:59.052470 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:44:59.052525 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:44:59.052542 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:44:59.053758 7405 raft_consensus.cc:1261] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab [term 1 FOLLOWER]: Refusing update from remote peer 59c6a5659bf64cef8715f0d82c50175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:59.053912 7678 raft_consensus.cc:1261] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Refusing update from remote peer 59c6a5659bf64cef8715f0d82c50175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:44:59.054051 7826 consensus_queue.cc:1035] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad25693f8c04a6997fa58bf2991aaab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:59.054190 7817 consensus_queue.cc:1035] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee24594ebeba400aa5a000075698796b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:44:59.055569 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.055458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:43113)
0928 22:44:59.055504 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.055534 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 22:44:59.055859 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.055766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:36587)
0928 22:44:59.055800 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.055826 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 22:44:59.056371 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.056277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:46627)
0928 22:44:59.056315 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.056340 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:44:59.056813 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.056543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.2:58095)
0928 22:44:59.056764 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.056793 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260928 22:44:59.057046 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.056936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.3:58263)
0928 22:44:59.056997 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.057019 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:44:59.057633 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:44:59.057546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.98.1:45411)
0928 22:44:59.057582 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.057604 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 22:44:59.058471 7270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:59.058559 7270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:44:59.059726 7270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aecc4c8c3bcf4d55b97bc5190a6f62d7: Sending DeleteTablet for 3 replicas of tablet 082ff898b2354e688e801a50ac191152
I20260928 22:44:59.060015 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet 082ff898b2354e688e801a50ac191152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47146
I20260928 22:44:59.060200 7839 tablet_replica.cc:330] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: stopping tablet replica
I20260928 22:44:59.060184 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet 082ff898b2354e688e801a50ac191152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33640
I20260928 22:44:59.060254 7839 raft_consensus.cc:2229] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:44:59.060338 7839 raft_consensus.cc:2258] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:59.060536 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 082ff898b2354e688e801a50ac191152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:44:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53496
I20260928 22:44:59.060667 7839 ts_tablet_manager.cc:1905] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:59.060708 7840 tablet_replica.cc:330] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: stopping tablet replica
I20260928 22:44:59.060781 7840 raft_consensus.cc:2229] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:44:59.060827 7840 raft_consensus.cc:2258] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:44:59.060870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7234
I20260928 22:44:59.060925 7841 tablet_replica.cc:330] T 082ff898b2354e688e801a50ac191152 P dad25693f8c04a6997fa58bf2991aaab: stopping tablet replica
I20260928 22:44:59.061718 7840 ts_tablet_manager.cc:1905] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:44:59.062894 7839 ts_tablet_manager.cc:1918] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:59.062955 7839 log.cc:1196] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/082ff898b2354e688e801a50ac191152
I20260928 22:44:59.063191 7839 ts_tablet_manager.cc:1939] T 082ff898b2354e688e801a50ac191152 P 59c6a5659bf64cef8715f0d82c50175a: Deleting consensus metadata
I20260928 22:44:59.063195 7840 ts_tablet_manager.cc:1918] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:44:59.063236 7840 log.cc:1196] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/082ff898b2354e688e801a50ac191152
I20260928 22:44:59.063446 7840 ts_tablet_manager.cc:1939] T 082ff898b2354e688e801a50ac191152 P ee24594ebeba400aa5a000075698796b: Deleting consensus metadata
I20260928 22:44:59.063494 7258 catalog_manager.cc:4841] TS 59c6a5659bf64cef8715f0d82c50175a (127.0.98.2:43117): tablet 082ff898b2354e688e801a50ac191152 (table test-table [id=ee61752682034ab9b35d29a6293b68be]) successfully deleted
I20260928 22:44:59.063783 7256 catalog_manager.cc:4841] TS ee24594ebeba400aa5a000075698796b (127.0.98.3:39077): tablet 082ff898b2354e688e801a50ac191152 (table test-table [id=ee61752682034ab9b35d29a6293b68be]) successfully deleted
W20260928 22:44:59.068960 7809 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:44:59.068761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46165 (local address 127.0.0.1:53504)
0928 22:44:59.068809 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:44:59.068819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:44:59.068863 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:44:59.068910 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46165: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:44:59.069098 7259 catalog_manager.cc:4584] TS dad25693f8c04a6997fa58bf2991aaab (127.0.98.1:46165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 082ff898b2354e688e801a50ac191152: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46165: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 22:44:59.069164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7459
I20260928 22:44:59.073860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7595
I20260928 22:44:59.078680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 6953
I20260928 22:44:59.082600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7140
2026-09-28T22:44:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4653 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:44:59 dist-test-slave-pxr1 krb5kdc[7844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:44:59 dist-test-slave-pxr1 krb5kdc[7844](info): set up 2 sockets
Sep 28 22:44:59 dist-test-slave-pxr1 krb5kdc[7844](info): commencing operation
krb5kdc: starting...
W20260928 22:45:00.029067 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 22:45:00.143767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:00.223353 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:00.223588 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:00.223611 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:00.224824 7857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:00.224866 7857 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:45:00.224879 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:00.224890 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:00.224901 7857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:00.224915 7857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:00.226404 7857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33633
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:00.226616 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:00.226791 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:00.231318 7862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:00.231446 7865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.231549 7857 server_base.cc:1034] running on GCE node
W20260928 22:45:00.231331 7863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.231930 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:00.232177 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:00.233305 7857 hybrid_clock.cc:648] HybridClock initialized: now 1790635500233294 us; error 28 us; skew 500 ppm
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:00.235646 7857 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 22:45:00.236032 7857 webserver.cc:460] Webserver started at http://127.0.98.62:44579/ using document root <none> and password file <none>
I20260928 22:45:00.236229 7857 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:00.236275 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:00.236402 7857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:00.237458 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "b1df3bb0268a4e09a9146dffc9c37897"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "1e9298ca2332abdbc52e5458b0995b1a"
server_key_iv: "1c1242dd56f21ae6895bd14551730bf6"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.237789 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1df3bb0268a4e09a9146dffc9c37897"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "1e9298ca2332abdbc52e5458b0995b1a"
server_key_iv: "1c1242dd56f21ae6895bd14551730bf6"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.239006 7857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:45:00.239635 7872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.239981 7857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:00.240072 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "b1df3bb0268a4e09a9146dffc9c37897"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "1e9298ca2332abdbc52e5458b0995b1a"
server_key_iv: "1c1242dd56f21ae6895bd14551730bf6"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.240144 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:00.248291 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:00.250255 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:00.250383 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:00.254521 7857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33633
I20260928 22:45:00.254581 7934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33633 every 8 connection(s)
I20260928 22:45:00.254865 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:00.255470 7935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.257714 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Bootstrap starting.
I20260928 22:45:00.258261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7857
I20260928 22:45:00.258294 7935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.258355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:45:00.258550 7935 log.cc:824] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:00.258634 392 external_mini_cluster.cc:1428] Setting key 34b8b2e0091881f1ef047e729ab37130
I20260928 22:45:00.259286 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: No bootstrap required, opened a new log
I20260928 22:45:00.261901 7935 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } }
I20260928 22:45:00.262081 7935 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.262118 7935 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1df3bb0268a4e09a9146dffc9c37897, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.262279 7935 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } }
I20260928 22:45:00.262363 7935 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:00.262393 7935 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:00.262441 7935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.263058 7935 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } }
I20260928 22:45:00.263164 7935 leader_election.cc:302] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1df3bb0268a4e09a9146dffc9c37897; no voters:
I20260928 22:45:00.263329 7935 leader_election.cc:288] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:45:00.263378 7940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:00.263520 7940 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [term 1 LEADER]: Becoming Leader. State: Replica: b1df3bb0268a4e09a9146dffc9c37897, State: Running, Role: LEADER
I20260928 22:45:00.263592 7935 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:00.263646 7940 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } }
I20260928 22:45:00.264048 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1df3bb0268a4e09a9146dffc9c37897. Latest consensus state: current_term: 1 leader_uuid: "b1df3bb0268a4e09a9146dffc9c37897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } } }
I20260928 22:45:00.264199 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:00.264071 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1df3bb0268a4e09a9146dffc9c37897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df3bb0268a4e09a9146dffc9c37897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33633 } } }
I20260928 22:45:00.264448 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:00.264918 7949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:00.265628 7949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:00.267198 7949 catalog_manager.cc:1347] Generated new cluster ID: 0451d803079f4faba2ed8d6467cd2f6a
I20260928 22:45:00.267246 7949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:00.269436 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.259105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.0.98.62:33633)
0928 22:45:00.259280 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.259290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.259325 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:45:00.259829 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.259839 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.259907 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.260073 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.260627 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.262073 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.262898 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.263531 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.263677 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.266293 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.266297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.266298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.266310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.268030 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.268351 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.268354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.268354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.268439 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.268614 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.268615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.268615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.268899 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.269060 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.269135 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.269263 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 22:45:00.288302 7949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:45:00.288934 7949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:45:00.296306 7949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Generated new TSK 0
I20260928 22:45:00.296521 7949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 22:45:00.331991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33633
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:00.421288 7961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:00.421526 7961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:00.421551 7961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:00.422766 7961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:00.422820 7961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:00.422840 7961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:45:00.422857 7961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:00.424623 7961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:00.424916 7961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:00.425163 7961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:00.430281 7967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:00.430307 7969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.430352 7961 server_base.cc:1034] running on GCE node
W20260928 22:45:00.430294 7966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.430652 7961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:00.430912 7961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:00.432061 7961 hybrid_clock.cc:648] HybridClock initialized: now 1790635500432049 us; error 31 us; skew 500 ppm
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:00.434703 7961 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 22:45:00.435153 7961 webserver.cc:460] Webserver started at http://127.0.98.1:37931/ using document root <none> and password file <none>
I20260928 22:45:00.435366 7961 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:00.435415 7961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:00.435536 7961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:00.436578 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "dabee8b16d274f24a899605a461d5985"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "78dc3fbb083d9fea4280763008d00355"
server_key_iv: "9ec5778dd1fbe6980f3ec2d4d794468d"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.436929 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dabee8b16d274f24a899605a461d5985"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "78dc3fbb083d9fea4280763008d00355"
server_key_iv: "9ec5778dd1fbe6980f3ec2d4d794468d"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.438145 7961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:00.438817 7976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.438962 7961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:00.439041 7961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "dabee8b16d274f24a899605a461d5985"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "78dc3fbb083d9fea4280763008d00355"
server_key_iv: "9ec5778dd1fbe6980f3ec2d4d794468d"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.439103 7961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:00.458964 7961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:00.460765 7961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:00.460903 7961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:00.461387 7961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:00.461431 7961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.461468 7961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:00.461493 7961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:00.467937 7961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41785
I20260928 22:45:00.468014 8091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41785 every 8 connection(s)
I20260928 22:45:00.468286 7961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:00.470261 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.461835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46443 (local address 127.0.98.62:33633)
0928 22:45:00.461896 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.461898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.462069 (+ 171us) server_negotiation.cc:407] Connection header received
0928 22:45:00.462554 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.462555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.462567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.462641 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.463409 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.464220 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.465456 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.465765 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.465833 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.467876 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.467878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.467878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.467886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.469242 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.469588 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.469589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.469589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.469639 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.469809 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.469810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.469810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.469902 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.469943 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.470142 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.470200 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:45:00.470434 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.461796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.1:46443)
0928 22:45:00.461946 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.461987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.462459 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.462708 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.462711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.463035 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.463311 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.463318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.464332 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.464333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.465380 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.465383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.465865 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.465865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.465976 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.466442 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.466451 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.467793 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.469308 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.469312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.469313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.469532 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.469682 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.469683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.469683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.469769 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.469942 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.469944 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.470053 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.470127 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.470239 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260928 22:45:00.470866 8092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33633
I20260928 22:45:00.470940 8092 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:00.471109 8092 heartbeater.cc:511] Master 127.0.98.62:33633 requested a full tablet report, sending...
I20260928 22:45:00.471616 7889 ts_manager.cc:194] Registered new tserver with Master: dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785)
I20260928 22:45:00.472226 7889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46443
I20260928 22:45:00.476441 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.471786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36857 (local address 127.0.98.62:33633)
0928 22:45:00.471822 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.471824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.471925 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:45:00.471968 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.471969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.471979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.472022 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.472286 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.472834 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.473547 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.473741 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.473791 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.474558 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.474559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.474560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.474566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.475495 (+ 929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.475771 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.475772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.475773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.475823 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.475987 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.475988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.475989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.476089 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.476141 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.476284 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.476348 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:45:00.476462 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.471738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.1:36857)
0928 22:45:00.471865 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.471874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.471923 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.472024 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.472024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.472175 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.472227 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.472228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.472937 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.472938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.473475 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.473477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.473797 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.473798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.473862 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.474187 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.474190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.474484 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.475565 (+ 1081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.475566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.475566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.475714 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.475876 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.475877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.475877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.475929 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.476142 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.476143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.476198 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.476254 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.476350 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260928 22:45:00.477264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 7961
I20260928 22:45:00.477384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:45:00.477602 392 external_mini_cluster.cc:1428] Setting key 52f615912217b5c068aa5c1a22fa297f
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 22:45:00.499548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33633
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:00.578092 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:00.578308 8101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:00.578341 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:00.579519 8101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:00.579564 8101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:00.579592 8101 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:45:00.579622 8101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:00.581161 8101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:00.581393 8101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:00.581609 8101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:00.586472 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.586527 8101 server_base.cc:1034] running on GCE node
W20260928 22:45:00.586467 8109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:00.586441 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.586814 8101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:00.587023 8101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:00.588163 8101 hybrid_clock.cc:648] HybridClock initialized: now 1790635500588150 us; error 29 us; skew 500 ppm
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:00.590648 8101 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 22:45:00.591089 8101 webserver.cc:460] Webserver started at http://127.0.98.2:45581/ using document root <none> and password file <none>
I20260928 22:45:00.591284 8101 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:00.591331 8101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:00.591459 8101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:00.592475 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8159f2489a24d47a64c90ef74abf334"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "3b98a636ace49c2d2979f839f5e2ae75"
server_key_iv: "124a5c91d60e33addaff7b80ae4a35b2"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.592839 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8159f2489a24d47a64c90ef74abf334"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "3b98a636ace49c2d2979f839f5e2ae75"
server_key_iv: "124a5c91d60e33addaff7b80ae4a35b2"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.594115 8101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:00.594841 8116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.594997 8101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:00.595077 8101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "f8159f2489a24d47a64c90ef74abf334"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "3b98a636ace49c2d2979f839f5e2ae75"
server_key_iv: "124a5c91d60e33addaff7b80ae4a35b2"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.595139 8101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:00.605927 8101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:00.607986 8101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:00.608124 8101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:00.608639 8101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:00.608678 8101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.608707 8101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:00.608745 8101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:00.615545 8101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39075
I20260928 22:45:00.615612 8231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39075 every 8 connection(s)
I20260928 22:45:00.615890 8101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:45:00.617859 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.609054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36483 (local address 127.0.98.62:33633)
0928 22:45:00.609100 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.609103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.609331 (+ 228us) server_negotiation.cc:407] Connection header received
0928 22:45:00.609803 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.609804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.609818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.609882 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.610540 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.611316 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.612305 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.612590 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.612666 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.615033 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.615034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.615035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.615042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.616451 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.616857 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.616858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.616858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.616916 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.617127 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.617128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.617128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.617242 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.617325 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.617679 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.617766 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:45:00.618023 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.609054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.2:36483)
0928 22:45:00.609183 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.609218 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.609710 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.609962 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.609966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.610202 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.610447 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.610452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.611406 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.611408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.612233 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.612236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.612672 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.612673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.612835 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.613331 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.613344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.614941 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.616533 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.616539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.616541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.616790 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.616968 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.616969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.616969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.617075 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.617318 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.617331 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.617493 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.617606 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.617786 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260928 22:45:00.618525 8232 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33633
I20260928 22:45:00.618606 8232 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:00.618803 8232 heartbeater.cc:511] Master 127.0.98.62:33633 requested a full tablet report, sending...
I20260928 22:45:00.619221 7888 ts_manager.cc:194] Registered new tserver with Master: f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075)
I20260928 22:45:00.619733 7888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36483
I20260928 22:45:00.623874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8101
I20260928 22:45:00.623993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:45:00.624128 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.619511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.2:38931)
0928 22:45:00.619629 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.619640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.619688 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.619799 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.619800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.619983 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.620048 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.620050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.620635 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.620636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.621178 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.621180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.621462 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.621463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.621509 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.621843 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.621846 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.622130 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.623279 (+ 1149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.623280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.623280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.623434 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.623568 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.623569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.623569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.623609 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.623807 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.623808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.623857 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.623930 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.624032 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260928 22:45:00.624138 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.619553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38931 (local address 127.0.98.62:33633)
0928 22:45:00.619626 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.619628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.619680 (+ 52us) server_negotiation.cc:407] Connection header received
0928 22:45:00.619725 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.619726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.619735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.619793 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.620116 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.620557 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.621242 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.621413 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.621459 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.622193 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.622194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.622195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.622200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.623220 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.623487 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.623487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.623488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.623529 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.623639 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.623639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.623639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.623756 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.623800 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.623964 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.624052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:45:00.624209 392 external_mini_cluster.cc:1428] Setting key 11b28c1c86ceb6070353d213dfc8845f
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 22:45:00.647640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33633
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:00.728989 8241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:00.729206 8241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:00.729230 8241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:00.730382 8241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:00.730419 8241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:00.730439 8241 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:45:00.730455 8241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:00.731894 8241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:00.732087 8241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:00.732275 8241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:00.737100 8247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.737187 8241 server_base.cc:1034] running on GCE node
W20260928 22:45:00.737097 8246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:00.737123 8249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:00.737459 8241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:00.737703 8241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:00.738868 8241 hybrid_clock.cc:648] HybridClock initialized: now 1790635500738830 us; error 52 us; skew 500 ppm
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:00.741562 8241 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 22:45:00.742036 8241 webserver.cc:460] Webserver started at http://127.0.98.3:41325/ using document root <none> and password file <none>
I20260928 22:45:00.742261 8241 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:00.742309 8241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:00.742435 8241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:00.743530 8241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "e3bbf149f139739e0142e6deecf2d159"
server_key_iv: "561d9fd82b185e3c049565ec1727886b"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.743882 8241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "e3bbf149f139739e0142e6deecf2d159"
server_key_iv: "561d9fd82b185e3c049565ec1727886b"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.745324 8241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:45:00.746511 8256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.746788 8241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:00.746948 8241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
format_stamp: "Formatted at 2026-09-28 22:45:00 on dist-test-slave-pxr1"
server_key: "e3bbf149f139739e0142e6deecf2d159"
server_key_iv: "561d9fd82b185e3c049565ec1727886b"
server_key_version: "encryptionkey@0"
I20260928 22:45:00.747054 8241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:00.773207 8241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:00.775116 8241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:00.775252 8241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:00.775751 8241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:00.775791 8241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.775823 8241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:00.775841 8241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.782423 8241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33577
I20260928 22:45:00.782549 8371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33577 every 8 connection(s)
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:00.782795 8241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:45:00.785676 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.776230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35691 (local address 127.0.98.62:33633)
0928 22:45:00.776297 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.776299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.776520 (+ 221us) server_negotiation.cc:407] Connection header received
0928 22:45:00.777227 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.777228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.777238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.777291 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.778157 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.778912 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.780389 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.780691 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.780797 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.783419 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.783421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.783421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.783437 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.784540 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.784951 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.784951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.784952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.785002 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.785182 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.785182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.785182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.785283 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.785339 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.785550 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.785611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:45:00.785869 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.776213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.3:35691)
0928 22:45:00.776376 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.776428 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.777124 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.777373 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.777377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.777739 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.778056 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.778062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.779003 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.779004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.780303 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.780305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.780806 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.780807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.780964 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.781466 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.781476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.783258 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.784611 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.784617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.784619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.784887 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.785051 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.785051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.785052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.785146 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.785338 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.785341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.785456 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.785538 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.785657 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 22:45:00.786310 8372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33633
I20260928 22:45:00.786386 8372 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:00.786554 8372 heartbeater.cc:511] Master 127.0.98.62:33633 requested a full tablet report, sending...
I20260928 22:45:00.786908 7888 ts_manager.cc:194] Registered new tserver with Master: 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:00.787277 7888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35691
I20260928 22:45:00.791986 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.787319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33201 (local address 127.0.98.62:33633)
0928 22:45:00.787362 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.787364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.787407 (+ 43us) server_negotiation.cc:407] Connection header received
0928 22:45:00.787464 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.787465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.787474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.787517 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.787798 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.788257 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.788987 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.789177 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.789226 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.790046 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.790047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.790048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.790054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.791005 (+ 951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.791291 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.791291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.791292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.791368 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.791527 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.791528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.791529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.791634 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.791685 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.791819 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.791892 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:45:00.792001 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.787249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.98.3:33201)
0928 22:45:00.787356 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.787366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.787421 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.787519 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.787520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.787686 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:00.787737 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.787739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.788343 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.788344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.788907 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.788909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.789230 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.789231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.789291 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.789671 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:00.789675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:00.789951 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:00.791079 (+ 1128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.791080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.791081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.791232 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.791423 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:00.791423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:00.791424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:00.791479 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:00.791686 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:00.791686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:00.791729 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.791789 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.791894 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
I20260928 22:45:00.792474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8241
I20260928 22:45:00.792570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:45:00.792766 392 external_mini_cluster.cc:1428] Setting key c991db63db1359b42b68ccf4c6d8fb73
I20260928 22:45:00.793704 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:45:00.813689 7939 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:00.808787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.62:33633)
0928 22:45:00.808838 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.808840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.808867 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:45:00.808890 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.808891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.808899 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.808947 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.809466 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.809950 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.810631 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.810805 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.810847 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.813612 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56210: BlockingRecv error: recv got EOF from 127.0.0.1:56210 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 22:45:00 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:00.820222 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.814334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.0.98.62:33633)
0928 22:45:00.814391 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.814392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.814450 (+ 58us) server_negotiation.cc:407] Connection header received
0928 22:45:00.814494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.814495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.814502 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.814545 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:00.814941 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.815388 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.815984 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.816151 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.816191 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.818130 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:00.818132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:00.818132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:00.818139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:00.819242 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.819574 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.819575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.819575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.819629 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:00.819786 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:00.819786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:00.819786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:00.819887 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:00.819982 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.820064 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.820133 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:45:00.821626 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:45:00.823197 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:00.829571 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.0.1:35628)
0928 22:45:00.825618 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.825627 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.825677 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.826008 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.826011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.826013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.826076 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.826078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.827086 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.827087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.828332 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.828334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.829233 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.829234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.829321 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.829337 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.829412 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.829455 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":87,"threads_started":1}
I20260928 22:45:00.829566 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.0.98.2:39075)
0928 22:45:00.825857 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.825862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.825871 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:00.825904 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.825906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.825918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.825989 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.826152 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.827013 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.828400 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.829155 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.829233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.829316 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.829428 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.829466 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":65,"threads_started":1}
I20260928 22:45:00.829685 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41785 (local address 127.0.0.1:45156)
0928 22:45:00.825492 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.825530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.825588 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.826023 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.826024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.826026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.826092 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.826094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.827397 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.827398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.828446 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.828447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.829282 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.829283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.829350 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.829355 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.829382 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.829453 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":9,"thread_start_us":75,"threads_started":1}
I20260928 22:45:00.829773 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.0.98.1:41785)
0928 22:45:00.825868 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.825871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.825879 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:00.825910 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.825912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.825924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.826166 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.826223 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.827328 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.828511 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.829198 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.829529 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.829611 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.829637 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.829675 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260928 22:45:00.830570 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33577 (local address 127.0.0.1:51718)
0928 22:45:00.825749 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.825757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.825789 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.826056 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.826056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.826058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.826115 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.826116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.828513 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.828514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.829216 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.829218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.830341 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.830342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.830403 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.830407 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.830469 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.830507 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260928 22:45:00.830453 8026 tablet_service.cc:1467] Processing CreateTablet for tablet 02cd596c0a9342f6a09052037c0fefda (DEFAULT_TABLE table=test-table [id=53da6e76e28340349ba1aa429efba66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.830605 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.825664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.98.3:33577)
0928 22:45:00.825835 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.825840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.825851 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:00.825903 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.825906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.825921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.826001 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.826183 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.827318 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.829302 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.830260 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.830329 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.830397 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.830483 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.830516 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 22:45:00.830533 8166 tablet_service.cc:1467] Processing CreateTablet for tablet 02cd596c0a9342f6a09052037c0fefda (DEFAULT_TABLE table=test-table [id=53da6e76e28340349ba1aa429efba66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.830823 8026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02cd596c0a9342f6a09052037c0fefda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.830852 8166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02cd596c0a9342f6a09052037c0fefda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.831441 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 02cd596c0a9342f6a09052037c0fefda (DEFAULT_TABLE table=test-table [id=53da6e76e28340349ba1aa429efba66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.831754 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02cd596c0a9342f6a09052037c0fefda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.833926 8400 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Bootstrap starting.
I20260928 22:45:00.834550 8400 tablet_bootstrap.cc:654] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.834672 8401 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Bootstrap starting.
I20260928 22:45:00.834986 8400 log.cc:824] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:00.835139 8402 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Bootstrap starting.
I20260928 22:45:00.835279 8401 tablet_bootstrap.cc:654] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.835593 8401 log.cc:824] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:00.835644 8402 tablet_bootstrap.cc:654] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.835911 8400 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: No bootstrap required, opened a new log
I20260928 22:45:00.835924 8402 log.cc:824] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:00.835971 8400 ts_tablet_manager.cc:1397] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:00.836320 8401 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: No bootstrap required, opened a new log
I20260928 22:45:00.836388 8401 ts_tablet_manager.cc:1397] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:00.836558 8402 tablet_bootstrap.cc:492] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: No bootstrap required, opened a new log
I20260928 22:45:00.836613 8402 ts_tablet_manager.cc:1397] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:00.837894 8400 raft_consensus.cc:348] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838065 8400 raft_consensus.cc:374] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.838099 8400 raft_consensus.cc:729] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.838213 8400 consensus_queue.cc:260] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838248 8401 raft_consensus.cc:348] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838390 8401 raft_consensus.cc:374] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.838282 8402 raft_consensus.cc:348] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838431 8401 raft_consensus.cc:729] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabee8b16d274f24a899605a461d5985, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.838436 8402 raft_consensus.cc:374] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.838469 8402 raft_consensus.cc:729] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.838478 8400 ts_tablet_manager.cc:1428] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:45:00.838560 8401 consensus_queue.cc:260] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838577 8402 consensus_queue.cc:260] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.838625 8232 heartbeater.cc:503] Master 127.0.98.62:33633 was elected leader, sending a full tablet report...
I20260928 22:45:00.838804 8401 ts_tablet_manager.cc:1428] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:45:00.838824 8372 heartbeater.cc:503] Master 127.0.98.62:33633 was elected leader, sending a full tablet report...
I20260928 22:45:00.838907 8092 heartbeater.cc:503] Master 127.0.98.62:33633 was elected leader, sending a full tablet report...
I20260928 22:45:00.839403 8402 ts_tablet_manager.cc:1428] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260928 22:45:00.866441 8233 tablet.cc:2369] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:45:00.903057 8407 raft_consensus.cc:482] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:00.903200 8407 raft_consensus.cc:504] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.903590 8407 leader_election.cc:288] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075)
I20260928 22:45:00.906982 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.903828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39965 (local address 127.0.98.2:39075)
0928 22:45:00.903870 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.903872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.904075 (+ 203us) server_negotiation.cc:407] Connection header received
0928 22:45:00.904122 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.904124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.904136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.904181 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.904384 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.905107 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906295 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.906653 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906711 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.906721 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.906860 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.906911 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:45:00.907131 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.903992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60901 (local address 127.0.98.1:41785)
0928 22:45:00.904044 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.904045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.904054 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:00.904080 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.904081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.904092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.904140 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.904287 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.904852 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906129 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.906711 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906784 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.906794 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.907028 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.907076 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:45:00.906985 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.98.3:39965)
0928 22:45:00.903995 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.904005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.904071 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.904222 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.904223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.904225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.904304 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.904306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.905189 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.905190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.906219 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.906221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.906725 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.906797 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.906801 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.906852 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.906889 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":93,"thread_start_us":96,"threads_started":1}
I20260928 22:45:00.906998 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41785 (local address 127.0.98.3:60901)
0928 22:45:00.903789 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.903800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.903838 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.904154 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.904155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.904157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.904225 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.904226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.904929 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.904930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.906038 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.906041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.906779 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.906779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.906849 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.906851 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.906893 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.906933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:45:00.907558 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02cd596c0a9342f6a09052037c0fefda" candidate_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985" is_pre_election: true
I20260928 22:45:00.907580 8186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02cd596c0a9342f6a09052037c0fefda" candidate_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8159f2489a24d47a64c90ef74abf334" is_pre_election: true
I20260928 22:45:00.907716 8046 raft_consensus.cc:2454] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aca0291dfed4b28b5a06fc85ff2c7a9 in term 0.
I20260928 22:45:00.907716 8186 raft_consensus.cc:2454] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aca0291dfed4b28b5a06fc85ff2c7a9 in term 0.
I20260928 22:45:00.907934 8260 leader_election.cc:302] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aca0291dfed4b28b5a06fc85ff2c7a9, dabee8b16d274f24a899605a461d5985; no voters:
I20260928 22:45:00.908069 8407 raft_consensus.cc:2781] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:00.908151 8407 raft_consensus.cc:482] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:00.908176 8407 raft_consensus.cc:3037] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.909360 8407 raft_consensus.cc:504] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.909528 8407 leader_election.cc:288] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [CANDIDATE]: Term 1 election: Requested vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075)
I20260928 22:45:00.909667 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02cd596c0a9342f6a09052037c0fefda" candidate_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985"
I20260928 22:45:00.909668 8186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02cd596c0a9342f6a09052037c0fefda" candidate_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8159f2489a24d47a64c90ef74abf334"
I20260928 22:45:00.909745 8046 raft_consensus.cc:3037] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.909746 8186 raft_consensus.cc:3037] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.910437 8186 raft_consensus.cc:2454] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aca0291dfed4b28b5a06fc85ff2c7a9 in term 1.
I20260928 22:45:00.910601 8046 raft_consensus.cc:2454] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aca0291dfed4b28b5a06fc85ff2c7a9 in term 1.
I20260928 22:45:00.910609 8257 leader_election.cc:302] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aca0291dfed4b28b5a06fc85ff2c7a9, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:00.910684 8407 raft_consensus.cc:2781] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:00.910826 8407 raft_consensus.cc:686] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 LEADER]: Becoming Leader. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Running, Role: LEADER
I20260928 22:45:00.910934 8407 consensus_queue.cc:237] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.911799 7887 catalog_manager.cc:5495] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } health_report { overall_health: HEALTHY } } }
I20260928 22:45:00.925683 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.922859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.0.98.3:33577)
0928 22:45:00.922910 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.922913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.922924 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:00.922995 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.922996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.923020 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.923093 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:00.923187 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.923945 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.924797 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.925118 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.925194 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.925333 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:00.925471 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:00.925552 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.925569 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.925594 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:45:00.927704 8186 raft_consensus.cc:1261] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Refusing update from remote peer 4aca0291dfed4b28b5a06fc85ff2c7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:00.927716 8046 raft_consensus.cc:1261] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Refusing update from remote peer 4aca0291dfed4b28b5a06fc85ff2c7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:00.928048 8407 consensus_queue.cc:1035] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:00.928190 8410 consensus_queue.cc:1035] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:00.930553 8413 mvcc.cc:205] Tried to move back new op lower bound from 7334443011796856832 to 7334443011731808256. Current Snapshot: MvccSnapshot[applied={T|T < 7334443011796856832}]
I20260928 22:45:00.931879 8414 mvcc.cc:205] Tried to move back new op lower bound from 7334443011796856832 to 7334443011731808256. Current Snapshot: MvccSnapshot[applied={T|T < 7334443011796856832}]
I20260928 22:45:00.931985 8415 mvcc.cc:205] Tried to move back new op lower bound from 7334443011796856832 to 7334443011731808256. Current Snapshot: MvccSnapshot[applied={T|T < 7334443011796856832 or (T in {7334443011796856832})}]
I20260928 22:45:00.933254 7889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:00.933357 7889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:00.934060 7889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Sending DeleteTablet for 3 replicas of tablet 02cd596c0a9342f6a09052037c0fefda
I20260928 22:45:00.934433 8026 tablet_service.cc:1514] Processing DeleteTablet for tablet 02cd596c0a9342f6a09052037c0fefda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45156
I20260928 22:45:00.934508 8166 tablet_service.cc:1514] Processing DeleteTablet for tablet 02cd596c0a9342f6a09052037c0fefda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35628
I20260928 22:45:00.934582 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 02cd596c0a9342f6a09052037c0fefda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260928 22:45:00.934969 8423 tablet_replica.cc:330] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: stopping tablet replica
I20260928 22:45:00.935072 8423 raft_consensus.cc:2229] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:00.935038 7889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:45:00.935170 7889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:00.935178 8423 raft_consensus.cc:2258] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:00.936614 8026 tablet_service.cc:1467] Processing CreateTablet for tablet d337f385cfee415689dd599fdfa77d59 (DEFAULT_TABLE table=test-table [id=b9732f2839fa46549b311cff70ae8a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.936774 8026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d337f385cfee415689dd599fdfa77d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.936666 8423 ts_tablet_manager.cc:1905] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:00.938079 8401 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Bootstrap starting.
I20260928 22:45:00.938293 8306 tablet_service.cc:1467] Processing CreateTablet for tablet d337f385cfee415689dd599fdfa77d59 (DEFAULT_TABLE table=test-table [id=b9732f2839fa46549b311cff70ae8a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.938402 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d337f385cfee415689dd599fdfa77d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.938509 8401 tablet_bootstrap.cc:654] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.938782 8423 ts_tablet_manager.cc:1918] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:00.938894 8423 log.cc:1196] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/02cd596c0a9342f6a09052037c0fefda
I20260928 22:45:00.939224 8423 ts_tablet_manager.cc:1939] T 02cd596c0a9342f6a09052037c0fefda P f8159f2489a24d47a64c90ef74abf334: Deleting consensus metadata
I20260928 22:45:00.939677 8402 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Bootstrap starting.
I20260928 22:45:00.939903 7873 catalog_manager.cc:4841] TS f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075): tablet 02cd596c0a9342f6a09052037c0fefda (table test-table [id=53da6e76e28340349ba1aa429efba66c]) successfully deleted
I20260928 22:45:00.939981 8166 tablet_service.cc:1467] Processing CreateTablet for tablet d337f385cfee415689dd599fdfa77d59 (DEFAULT_TABLE table=test-table [id=b9732f2839fa46549b311cff70ae8a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:00.940071 8166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d337f385cfee415689dd599fdfa77d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:00.940083 8402 tablet_bootstrap.cc:654] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.940665 8402 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: No bootstrap required, opened a new log
I20260928 22:45:00.940696 8402 ts_tablet_manager.cc:1397] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:00.940847 8402 raft_consensus.cc:348] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.940897 8402 raft_consensus.cc:374] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.940908 8402 raft_consensus.cc:729] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.940943 8402 consensus_queue.cc:260] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.941020 8402 ts_tablet_manager.cc:1428] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.941179 8424 tablet_replica.cc:330] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: stopping tablet replica
I20260928 22:45:00.941218 8400 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Bootstrap starting.
I20260928 22:45:00.941275 8424 raft_consensus.cc:2229] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:45:00.941404 8424 raft_consensus.cc:2258] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:00.941581 8400 tablet_bootstrap.cc:654] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:00.942363 8400 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: No bootstrap required, opened a new log
I20260928 22:45:00.942407 8400 ts_tablet_manager.cc:1397] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:00.942543 8400 raft_consensus.cc:348] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.942601 8400 raft_consensus.cc:374] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.942619 8400 raft_consensus.cc:729] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.942667 8400 consensus_queue.cc:260] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.942746 8400 ts_tablet_manager.cc:1428] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.943073 8424 ts_tablet_manager.cc:1905] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:00.944617 8424 ts_tablet_manager.cc:1918] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:00.944759 8424 log.cc:1196] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/02cd596c0a9342f6a09052037c0fefda
I20260928 22:45:00.945037 8422 tablet_replica.cc:330] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: stopping tablet replica
I20260928 22:45:00.945050 8424 ts_tablet_manager.cc:1939] T 02cd596c0a9342f6a09052037c0fefda P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting consensus metadata
I20260928 22:45:00.945106 8401 tablet_bootstrap.cc:492] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: No bootstrap required, opened a new log
I20260928 22:45:00.945150 8422 raft_consensus.cc:2229] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:00.945152 8401 ts_tablet_manager.cc:1397] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 22:45:00.945307 8422 raft_consensus.cc:2258] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:00.945330 8401 raft_consensus.cc:348] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.945403 8401 raft_consensus.cc:374] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:00.945421 8401 raft_consensus.cc:729] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabee8b16d274f24a899605a461d5985, State: Initialized, Role: FOLLOWER
I20260928 22:45:00.945477 8401 consensus_queue.cc:260] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.945570 8401 ts_tablet_manager.cc:1428] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:00.946151 7876 catalog_manager.cc:4841] TS 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577): tablet 02cd596c0a9342f6a09052037c0fefda (table test-table [id=53da6e76e28340349ba1aa429efba66c]) successfully deleted
I20260928 22:45:00.946292 8422 ts_tablet_manager.cc:1905] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:00.947891 8422 ts_tablet_manager.cc:1918] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:00.948274 8422 log.cc:1196] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/02cd596c0a9342f6a09052037c0fefda
I20260928 22:45:00.948568 8422 ts_tablet_manager.cc:1939] T 02cd596c0a9342f6a09052037c0fefda P dabee8b16d274f24a899605a461d5985: Deleting consensus metadata
I20260928 22:45:00.948983 7876 catalog_manager.cc:4841] TS dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785): tablet 02cd596c0a9342f6a09052037c0fefda (table test-table [id=53da6e76e28340349ba1aa429efba66c]) successfully deleted
I20260928 22:45:00.964998 8406 raft_consensus.cc:482] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:00.965102 8406 raft_consensus.cc:504] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.965503 8406 leader_election.cc:288] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
W20260928 22:45:00.969189 8093 tablet.cc:2369] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:45:00.969370 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.965712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41785 (local address 127.0.98.2:51603)
0928 22:45:00.965757 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.965769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.965815 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.965993 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.965995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.965996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.966061 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.966063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.966989 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.966990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.968268 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.968271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.969061 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.969062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.969159 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.969165 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.969223 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.969272 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 22:45:00.970072 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.965762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51603 (local address 127.0.98.1:41785)
0928 22:45:00.965804 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.965806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.965816 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:00.965873 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.965874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.965882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.965944 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.966129 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.966887 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.968376 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.968984 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.969060 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.969072 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.969934 (+ 862us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.970002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:45:00.970733 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d337f385cfee415689dd599fdfa77d59" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985" is_pre_election: true
I20260928 22:45:00.970827 8046 raft_consensus.cc:2454] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:00.970997 8120 leader_election.cc:302] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:00.971120 8406 raft_consensus.cc:2781] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:00.971146 8406 raft_consensus.cc:482] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:00.971164 8406 raft_consensus.cc:3037] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.972043 8406 raft_consensus.cc:504] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.972178 8406 leader_election.cc:288] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Requested vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:00.972391 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d337f385cfee415689dd599fdfa77d59" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985"
I20260928 22:45:00.972462 8046 raft_consensus.cc:3037] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.973021 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.965816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33577 (local address 127.0.98.2:44871)
0928 22:45:00.969448 (+ 3632us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:00.969459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:00.969517 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:00.969682 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:00.969683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:00.969685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:00.969721 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.969722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.970602 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.970602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.971852 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:00.971856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.972763 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.972764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:00.972833 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.972838 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:45:00.972889 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:45:00.972928 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3619,"thread_start_us":64,"threads_started":1}
I20260928 22:45:00.973228 8046 raft_consensus.cc:2454] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
I20260928 22:45:00.973332 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:00.966022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44871 (local address 127.0.98.3:33577)
0928 22:45:00.966064 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:00.966066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:00.969572 (+ 3506us) server_negotiation.cc:407] Connection header received
0928 22:45:00.969605 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:00.969606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:00.969616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:00.969665 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:00.969800 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.970506 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.971963 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:00.972519 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:00.972588 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:00.972600 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:00.973224 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 22:45:00.973267 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:45:00.973410 8120 leader_election.cc:302] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:00.973716 8406 raft_consensus.cc:2781] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:00.973861 8406 raft_consensus.cc:686] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 1 LEADER]: Becoming Leader. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Running, Role: LEADER
I20260928 22:45:00.973841 8326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d337f385cfee415689dd599fdfa77d59" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" is_pre_election: true
I20260928 22:45:00.973925 8406 consensus_queue.cc:237] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:00.973975 8326 raft_consensus.cc:2454] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:00.974090 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d337f385cfee415689dd599fdfa77d59" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
I20260928 22:45:00.974133 8325 raft_consensus.cc:3037] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:00.974694 7889 catalog_manager.cc:5495] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8159f2489a24d47a64c90ef74abf334 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } health_report { overall_health: UNKNOWN } } }
I20260928 22:45:00.974882 8325 raft_consensus.cc:2454] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:01.028158 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.021570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.98.62:33633)
0928 22:45:01.021626 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.021628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.021740 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:45:01.021818 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.021819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.021827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.021868 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:01.022310 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.022781 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.023450 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.023618 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.023670 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.025750 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:01.025752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:01.025752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:01.025758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:01.027059 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:01.027487 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:01.027488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:01.027488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:01.027532 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:01.027706 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:01.027706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:01.027706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:01.027800 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:01.027850 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.028010 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.028092 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":33}
I20260928 22:45:01.028282 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.021497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33633 (local address 127.0.0.1:56218)
0928 22:45:01.021667 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.021678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.021749 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.021885 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.021885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.022177 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:01.022236 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.022239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.022862 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.022863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.023382 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.023383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.023673 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.023674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.023720 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.024198 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:01.024202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:01.025664 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:01.027127 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:01.027134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:01.027137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:01.027417 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:01.027586 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:01.027587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:01.027587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:01.027654 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:01.027859 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:01.027865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:01.027923 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.027995 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.028158 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260928 22:45:01.030313 7889 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56218:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:45:01.030550 7889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:01.031831 8026 tablet_service.cc:1467] Processing CreateTablet for tablet 6b306b7a784b40f393eabcb2e1ec0679 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd9872cce01442c794d291929208e6db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:45:01.031870 8166 tablet_service.cc:1467] Processing CreateTablet for tablet 6b306b7a784b40f393eabcb2e1ec0679 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd9872cce01442c794d291929208e6db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:45:01.031987 8166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b306b7a784b40f393eabcb2e1ec0679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.031988 8026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b306b7a784b40f393eabcb2e1ec0679. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 22:45:01.033406 8373 tablet.cc:2369] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:45:01.033463 8400 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: Bootstrap starting.
I20260928 22:45:01.033463 8401 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: Bootstrap starting.
I20260928 22:45:01.033839 8400 tablet_bootstrap.cc:654] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.033843 8401 tablet_bootstrap.cc:654] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.034420 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 6b306b7a784b40f393eabcb2e1ec0679 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd9872cce01442c794d291929208e6db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:45:01.034552 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b306b7a784b40f393eabcb2e1ec0679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.034636 8401 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: No bootstrap required, opened a new log
I20260928 22:45:01.034679 8401 ts_tablet_manager.cc:1397] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:01.034798 8401 raft_consensus.cc:348] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.034847 8401 raft_consensus.cc:374] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.034858 8401 raft_consensus.cc:729] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabee8b16d274f24a899605a461d5985, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.034893 8401 consensus_queue.cc:260] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.034972 8401 ts_tablet_manager.cc:1428] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.035013 8408 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.035073 8408 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.035331 8400 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: No bootstrap required, opened a new log
I20260928 22:45:01.035372 8400 ts_tablet_manager.cc:1397] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:01.035490 8400 raft_consensus.cc:348] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.035532 8400 raft_consensus.cc:374] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.035543 8400 raft_consensus.cc:729] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.035578 8400 consensus_queue.cc:260] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.035651 8400 ts_tablet_manager.cc:1428] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.035722 8429 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.035792 8429 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.035957 8402 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Bootstrap starting.
I20260928 22:45:01.036342 8402 tablet_bootstrap.cc:654] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.036948 8402 tablet_bootstrap.cc:492] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: No bootstrap required, opened a new log
I20260928 22:45:01.037002 8402 ts_tablet_manager.cc:1397] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:01.037132 8402 raft_consensus.cc:348] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.037187 8402 raft_consensus.cc:374] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.037205 8402 raft_consensus.cc:729] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.037245 8402 consensus_queue.cc:260] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.037330 8402 ts_tablet_manager.cc:1428] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.037369 8407 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.037461 8407 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.109017 8429 raft_consensus.cc:482] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:01.109112 8429 raft_consensus.cc:504] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.109256 8429 leader_election.cc:288] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.109519 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b306b7a784b40f393eabcb2e1ec0679" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" is_pre_election: true
I20260928 22:45:01.109511 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b306b7a784b40f393eabcb2e1ec0679" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985" is_pre_election: true
I20260928 22:45:01.109603 8325 raft_consensus.cc:2454] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:01.109614 8046 raft_consensus.cc:2454] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:01.109756 8120 leader_election.cc:302] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:01.109856 8429 raft_consensus.cc:2781] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:01.109897 8429 raft_consensus.cc:482] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:01.109911 8429 raft_consensus.cc:3037] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.110544 8429 raft_consensus.cc:504] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.110646 8429 leader_election.cc:288] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Requested vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.110838 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b306b7a784b40f393eabcb2e1ec0679" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985"
I20260928 22:45:01.110865 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b306b7a784b40f393eabcb2e1ec0679" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
I20260928 22:45:01.110919 8046 raft_consensus.cc:3037] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.110922 8325 raft_consensus.cc:3037] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.111603 8325 raft_consensus.cc:2454] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
I20260928 22:45:01.111603 8046 raft_consensus.cc:2454] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
I20260928 22:45:01.111773 8120 leader_election.cc:302] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aca0291dfed4b28b5a06fc85ff2c7a9, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:01.111882 8429 raft_consensus.cc:2781] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:01.111936 8429 raft_consensus.cc:686] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [term 1 LEADER]: Becoming Leader. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Running, Role: LEADER
I20260928 22:45:01.111979 8429 consensus_queue.cc:237] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.112238 8406 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: TxnStatusTablet state changed. Reason: New leader f8159f2489a24d47a64c90ef74abf334. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.112313 8406 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:45:01.112491 8441 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:45:01.112605 8441 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:45:01.112582 7889 catalog_manager.cc:5495] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8159f2489a24d47a64c90ef74abf334 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } health_report { overall_health: UNKNOWN } } }
I20260928 22:45:01.131930 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.129365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.0.98.2:39075)
0928 22:45:01.129417 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.129419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.129427 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:01.129481 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.129481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.129497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.129550 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.129672 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.130149 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.131219 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.131421 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.131468 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.131650 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.131743 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.131790 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.131853 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.131870 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:45:01.131989 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.129255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.0.1:35634)
0928 22:45:01.129339 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.129353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.129406 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.129560 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.129561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.129562 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:45:01.129617 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.129620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.130219 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.130220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.131147 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.131150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.131487 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.131488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.131555 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.131573 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.131797 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.131802 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.131850 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.131897 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:45:01.133091 8046 raft_consensus.cc:1261] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.133152 8325 raft_consensus.cc:1261] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.133363 8429 consensus_queue.cc:1035] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.133496 8406 consensus_queue.cc:1035] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.134078 8408 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: TxnStatusTablet state changed. Reason: New leader f8159f2489a24d47a64c90ef74abf334. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.134107 8407 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: TxnStatusTablet state changed. Reason: New leader f8159f2489a24d47a64c90ef74abf334. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.134174 8408 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.134183 8407 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.134675 8406 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.134758 8406 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:45:01.134739 8408 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.134816 8408 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P dabee8b16d274f24a899605a461d5985: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.135452 8406 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.135531 8406 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P f8159f2489a24d47a64c90ef74abf334: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:45:01.135685 8407 tablet_replica.cc:439] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } } }
I20260928 22:45:01.135762 8407 tablet_replica.cc:442] T 6b306b7a784b40f393eabcb2e1ec0679 P 4aca0291dfed4b28b5a06fc85ff2c7a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:45:01.138671 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.136232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.0.98.2:39075)
0928 22:45:01.136300 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.136302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.136309 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:01.136332 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.136332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.136341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.136399 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.136478 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.137253 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.137980 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.138291 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.138352 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.138467 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.138502 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.138551 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.138584 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.138602 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:45:01.143821 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.141126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57755 (local address 127.0.98.2:39075)
0928 22:45:01.141164 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.141166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.141318 (+ 152us) server_negotiation.cc:407] Connection header received
0928 22:45:01.141357 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.141358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.141367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.141423 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.141551 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.142257 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.143091 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.143381 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.143440 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.143603 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.143634 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.143681 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.143718 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.143737 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:45:01.143834 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.141074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.98.2:57755)
0928 22:45:01.141257 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.141266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.141320 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.141426 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.141427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.141428 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:45:01.141501 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.141502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.142344 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.142345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.143023 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.143026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.143448 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.143449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.143503 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.143518 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.143684 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.143686 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.143715 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.143747 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260928 22:45:01.145993 8325 raft_consensus.cc:1261] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.146018 8046 raft_consensus.cc:1261] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.146204 8406 consensus_queue.cc:1035] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.146335 8449 consensus_queue.cc:1035] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.167636 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.167743 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.168378 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Sending DeleteTablet for 3 replicas of tablet d337f385cfee415689dd599fdfa77d59
I20260928 22:45:01.168651 8026 tablet_service.cc:1514] Processing DeleteTablet for tablet d337f385cfee415689dd599fdfa77d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45156
I20260928 22:45:01.168759 8166 tablet_service.cc:1514] Processing DeleteTablet for tablet d337f385cfee415689dd599fdfa77d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35628
I20260928 22:45:01.168787 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet d337f385cfee415689dd599fdfa77d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260928 22:45:01.168874 8422 tablet_replica.cc:330] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: stopping tablet replica
I20260928 22:45:01.168923 8422 raft_consensus.cc:2229] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.168958 8422 raft_consensus.cc:2258] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.169030 8423 tablet_replica.cc:330] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: stopping tablet replica
I20260928 22:45:01.169080 8423 raft_consensus.cc:2229] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:45:01.169180 8423 raft_consensus.cc:2258] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.169255 8424 tablet_replica.cc:330] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: stopping tablet replica
I20260928 22:45:01.169302 8424 raft_consensus.cc:2229] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.169334 8424 raft_consensus.cc:2258] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.169778 8422 ts_tablet_manager.cc:1905] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.171581 8422 ts_tablet_manager.cc:1918] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.171602 8423 ts_tablet_manager.cc:1905] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.171653 8422 log.cc:1196] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/d337f385cfee415689dd599fdfa77d59
I20260928 22:45:01.171849 8422 ts_tablet_manager.cc:1939] T d337f385cfee415689dd599fdfa77d59 P dabee8b16d274f24a899605a461d5985: Deleting consensus metadata
I20260928 22:45:01.172190 7876 catalog_manager.cc:4841] TS dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785): tablet d337f385cfee415689dd599fdfa77d59 (table test-table [id=b9732f2839fa46549b311cff70ae8a40]) successfully deleted
I20260928 22:45:01.172216 8424 ts_tablet_manager.cc:1905] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.172984 8423 ts_tablet_manager.cc:1918] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.173035 8423 log.cc:1196] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/d337f385cfee415689dd599fdfa77d59
I20260928 22:45:01.173223 8423 ts_tablet_manager.cc:1939] T d337f385cfee415689dd599fdfa77d59 P f8159f2489a24d47a64c90ef74abf334: Deleting consensus metadata
I20260928 22:45:01.173667 7873 catalog_manager.cc:4841] TS f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075): tablet d337f385cfee415689dd599fdfa77d59 (table test-table [id=b9732f2839fa46549b311cff70ae8a40]) successfully deleted
I20260928 22:45:01.173734 8424 ts_tablet_manager.cc:1918] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.173791 8424 log.cc:1196] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/d337f385cfee415689dd599fdfa77d59
I20260928 22:45:01.173977 8424 ts_tablet_manager.cc:1939] T d337f385cfee415689dd599fdfa77d59 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting consensus metadata
I20260928 22:45:01.174258 7876 catalog_manager.cc:4841] TS 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577): tablet d337f385cfee415689dd599fdfa77d59 (table test-table [id=b9732f2839fa46549b311cff70ae8a40]) successfully deleted
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:45:01.196156 7939 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:01.190546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.0.98.62:33633)
0928 22:45:01.190601 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.190603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.190683 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:45:01.190748 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.190749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.190761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.190818 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:01.191342 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.192064 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.192814 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.193097 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.193176 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.196089 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56232: BlockingRecv error: recv got EOF from 127.0.0.1:56232 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 22:45:01.204264 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.196929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.98.62:33633)
0928 22:45:01.196976 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.196979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.197007 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:45:01.197082 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.197083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.197090 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.197134 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:01.197807 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.198382 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.199364 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.199552 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.199597 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.202003 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:01.202005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:01.202005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:01.202012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:01.203264 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:01.203582 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:01.203583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:01.203583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:01.203640 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:01.203799 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:01.203799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:01.203800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:01.203908 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:01.204003 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.204094 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.204170 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:45:01.205789 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:45:01.205953 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:01.207350 8026 tablet_service.cc:1467] Processing CreateTablet for tablet 64342b91b6e3435792fae64fe668c413 (DEFAULT_TABLE table=test-table [id=e7fe3d2de1214e5da04d023439985ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.207507 8026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64342b91b6e3435792fae64fe668c413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.209052 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 64342b91b6e3435792fae64fe668c413 (DEFAULT_TABLE table=test-table [id=e7fe3d2de1214e5da04d023439985ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.209177 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64342b91b6e3435792fae64fe668c413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.210428 8402 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Bootstrap starting.
I20260928 22:45:01.210469 8401 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Bootstrap starting.
I20260928 22:45:01.210762 8402 tablet_bootstrap.cc:654] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.210863 8401 tablet_bootstrap.cc:654] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.211304 8402 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: No bootstrap required, opened a new log
I20260928 22:45:01.211295 8166 tablet_service.cc:1467] Processing CreateTablet for tablet 64342b91b6e3435792fae64fe668c413 (DEFAULT_TABLE table=test-table [id=e7fe3d2de1214e5da04d023439985ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.211347 8402 ts_tablet_manager.cc:1397] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:01.211416 8166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64342b91b6e3435792fae64fe668c413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.211467 8402 raft_consensus.cc:348] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.211529 8402 raft_consensus.cc:374] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.211547 8402 raft_consensus.cc:729] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.211585 8402 consensus_queue.cc:260] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.211632 8401 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: No bootstrap required, opened a new log
I20260928 22:45:01.211658 8402 ts_tablet_manager.cc:1428] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.211673 8401 ts_tablet_manager.cc:1397] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:01.211809 8401 raft_consensus.cc:348] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.211872 8401 raft_consensus.cc:374] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.211894 8401 raft_consensus.cc:729] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabee8b16d274f24a899605a461d5985, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.211936 8401 consensus_queue.cc:260] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.212021 8401 ts_tablet_manager.cc:1428] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.212612 8400 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Bootstrap starting.
I20260928 22:45:01.213088 8400 tablet_bootstrap.cc:654] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.213897 8400 tablet_bootstrap.cc:492] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: No bootstrap required, opened a new log
I20260928 22:45:01.213940 8400 ts_tablet_manager.cc:1397] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:01.214071 8400 raft_consensus.cc:348] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.214128 8400 raft_consensus.cc:374] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.214144 8400 raft_consensus.cc:729] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.214183 8400 consensus_queue.cc:260] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.214269 8400 ts_tablet_manager.cc:1428] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.217134 8449 raft_consensus.cc:482] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:01.217195 8449 raft_consensus.cc:504] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.217352 8449 leader_election.cc:288] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.217473 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64342b91b6e3435792fae64fe668c413" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985" is_pre_election: true
I20260928 22:45:01.217553 8046 raft_consensus.cc:2454] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:01.217506 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64342b91b6e3435792fae64fe668c413" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" is_pre_election: true
I20260928 22:45:01.217594 8325 raft_consensus.cc:2454] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 0.
I20260928 22:45:01.217695 8120 leader_election.cc:302] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:01.217775 8449 raft_consensus.cc:2781] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:01.217814 8449 raft_consensus.cc:482] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:01.217854 8449 raft_consensus.cc:3037] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.218456 8449 raft_consensus.cc:504] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.218559 8449 leader_election.cc:288] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Requested vote from peers dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.218688 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64342b91b6e3435792fae64fe668c413" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabee8b16d274f24a899605a461d5985"
I20260928 22:45:01.218755 8046 raft_consensus.cc:3037] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.218719 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64342b91b6e3435792fae64fe668c413" candidate_uuid: "f8159f2489a24d47a64c90ef74abf334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
I20260928 22:45:01.218788 8325 raft_consensus.cc:3037] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.219293 8046 raft_consensus.cc:2454] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
I20260928 22:45:01.219448 8120 leader_election.cc:302] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:01.219525 8449 raft_consensus.cc:2781] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:01.219558 8449 raft_consensus.cc:686] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 1 LEADER]: Becoming Leader. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Running, Role: LEADER
I20260928 22:45:01.219594 8449 consensus_queue.cc:237] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.219643 8325 raft_consensus.cc:2454] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8159f2489a24d47a64c90ef74abf334 in term 1.
I20260928 22:45:01.220120 7889 catalog_manager.cc:5495] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8159f2489a24d47a64c90ef74abf334 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f8159f2489a24d47a64c90ef74abf334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } health_report { overall_health: UNKNOWN } } }
I20260928 22:45:01.228560 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.226087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.98.2:39075)
0928 22:45:01.226124 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.226126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.226137 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:01.226209 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.226210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.226218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.226261 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.226384 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.226857 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.227960 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.228144 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.228196 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.228364 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.228390 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.228420 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.228473 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.228491 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:45:01.229238 8325 raft_consensus.cc:1261] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.229265 8046 raft_consensus.cc:1261] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Refusing update from remote peer f8159f2489a24d47a64c90ef74abf334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.229473 8406 consensus_queue.cc:1035] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.229617 8449 consensus_queue.cc:1035] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.232450 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.232534 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.233089 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Sending DeleteTablet for 3 replicas of tablet 64342b91b6e3435792fae64fe668c413
I20260928 22:45:01.233364 8166 tablet_service.cc:1514] Processing DeleteTablet for tablet 64342b91b6e3435792fae64fe668c413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35628
I20260928 22:45:01.233381 8026 tablet_service.cc:1514] Processing DeleteTablet for tablet 64342b91b6e3435792fae64fe668c413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45156
I20260928 22:45:01.233470 8422 tablet_replica.cc:330] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: stopping tablet replica
I20260928 22:45:01.233470 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 64342b91b6e3435792fae64fe668c413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260928 22:45:01.233511 8422 raft_consensus.cc:2229] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.233546 8422 raft_consensus.cc:2258] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.233549 8423 tablet_replica.cc:330] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: stopping tablet replica
I20260928 22:45:01.233493 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 22:45:01.233592 8423 raft_consensus.cc:2229] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 1 LEADER]: Raft consensus shutting down.
W20260928 22:45:01.233616 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:01.233641 8423 raft_consensus.cc:2258] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.233706 8424 tablet_replica.cc:330] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: stopping tablet replica
I20260928 22:45:01.233754 8424 raft_consensus.cc:2229] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.233786 8424 raft_consensus.cc:2258] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.234339 8423 ts_tablet_manager.cc:1905] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.234608 8424 ts_tablet_manager.cc:1905] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.235759 8422 ts_tablet_manager.cc:1905] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.235939 8423 ts_tablet_manager.cc:1918] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:01.235987 8423 log.cc:1196] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/64342b91b6e3435792fae64fe668c413
I20260928 22:45:01.236160 8423 ts_tablet_manager.cc:1939] T 64342b91b6e3435792fae64fe668c413 P f8159f2489a24d47a64c90ef74abf334: Deleting consensus metadata
I20260928 22:45:01.236236 8424 ts_tablet_manager.cc:1918] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:01.236284 8424 log.cc:1196] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/64342b91b6e3435792fae64fe668c413
I20260928 22:45:01.236475 8424 ts_tablet_manager.cc:1939] T 64342b91b6e3435792fae64fe668c413 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting consensus metadata
I20260928 22:45:01.236819 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 (DEFAULT_TABLE table=test-table [id=403753469af84d16870ad2f02325ec84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.236910 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861351ef1b54c50a79ea9e621a7d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.237185 8422 ts_tablet_manager.cc:1918] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:01.237228 8422 log.cc:1196] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/64342b91b6e3435792fae64fe668c413
I20260928 22:45:01.237402 8422 ts_tablet_manager.cc:1939] T 64342b91b6e3435792fae64fe668c413 P dabee8b16d274f24a899605a461d5985: Deleting consensus metadata
I20260928 22:45:01.237979 8402 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Bootstrap starting.
I20260928 22:45:01.238178 8166 tablet_service.cc:1467] Processing CreateTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 (DEFAULT_TABLE table=test-table [id=403753469af84d16870ad2f02325ec84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.238258 8166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861351ef1b54c50a79ea9e621a7d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.238317 8402 tablet_bootstrap.cc:654] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.238797 8402 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: No bootstrap required, opened a new log
I20260928 22:45:01.238837 8402 ts_tablet_manager.cc:1397] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:01.238943 8402 raft_consensus.cc:348] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.239005 8402 raft_consensus.cc:374] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.239024 8402 raft_consensus.cc:729] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aca0291dfed4b28b5a06fc85ff2c7a9, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.239069 8402 consensus_queue.cc:260] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.239154 8402 ts_tablet_manager.cc:1428] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.239217 8400 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Bootstrap starting.
I20260928 22:45:01.239574 8400 tablet_bootstrap.cc:654] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.239869 7873 catalog_manager.cc:4841] TS f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075): tablet 64342b91b6e3435792fae64fe668c413 (table test-table [id=e7fe3d2de1214e5da04d023439985ff1]) successfully deleted
I20260928 22:45:01.240075 8026 tablet_service.cc:1467] Processing CreateTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 (DEFAULT_TABLE table=test-table [id=403753469af84d16870ad2f02325ec84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:01.240170 8026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861351ef1b54c50a79ea9e621a7d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:01.240185 7876 catalog_manager.cc:4841] TS dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785): tablet 64342b91b6e3435792fae64fe668c413 (table test-table [id=e7fe3d2de1214e5da04d023439985ff1]) successfully deleted
I20260928 22:45:01.240232 7876 catalog_manager.cc:4841] TS 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577): tablet 64342b91b6e3435792fae64fe668c413 (table test-table [id=e7fe3d2de1214e5da04d023439985ff1]) successfully deleted
I20260928 22:45:01.240780 8400 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: No bootstrap required, opened a new log
I20260928 22:45:01.240828 8400 ts_tablet_manager.cc:1397] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:01.240965 8400 raft_consensus.cc:348] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.241024 8400 raft_consensus.cc:374] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.241040 8400 raft_consensus.cc:729] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8159f2489a24d47a64c90ef74abf334, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.241098 8400 consensus_queue.cc:260] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.241178 8400 ts_tablet_manager.cc:1428] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.241456 8401 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Bootstrap starting.
I20260928 22:45:01.242041 8401 tablet_bootstrap.cc:654] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:01.242624 8401 tablet_bootstrap.cc:492] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: No bootstrap required, opened a new log
I20260928 22:45:01.242663 8401 ts_tablet_manager.cc:1397] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:01.242790 8401 raft_consensus.cc:348] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.242847 8401 raft_consensus.cc:374] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:01.242863 8401 raft_consensus.cc:729] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabee8b16d274f24a899605a461d5985, State: Initialized, Role: FOLLOWER
I20260928 22:45:01.242904 8401 consensus_queue.cc:260] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.242998 8401 ts_tablet_manager.cc:1428] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:01.306257 8408 raft_consensus.cc:482] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:01.306363 8408 raft_consensus.cc:504] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.306707 8408 leader_election.cc:288] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.310062 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.306939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54767 (local address 127.0.98.2:39075)
0928 22:45:01.306986 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.306987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.307101 (+ 114us) server_negotiation.cc:407] Connection header received
0928 22:45:01.307217 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.307219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.307229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.307287 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:01.307449 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.308185 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.309309 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.309731 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.309791 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.309802 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.309953 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.309993 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:45:01.310405 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.307050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36135 (local address 127.0.98.3:33577)
0928 22:45:01.307122 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.307124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.307197 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:45:01.307273 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.307274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.307287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.307329 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:01.307495 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.308181 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.309515 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.310030 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.310072 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.310080 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.310301 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.310340 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:45:01.310420 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.306886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33577 (local address 127.0.98.1:36135)
0928 22:45:01.307147 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.307155 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.307215 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.307353 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.307354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.307356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:01.307433 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.307435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.308272 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.308272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.309394 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.309397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.310153 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.310154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.310229 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.310232 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.310277 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.310318 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":27,"thread_start_us":57,"threads_started":1}
I20260928 22:45:01.310626 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861351ef1b54c50a79ea9e621a7d325" candidate_uuid: "dabee8b16d274f24a899605a461d5985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" is_pre_election: true
I20260928 22:45:01.310426 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.306874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.98.1:54767)
0928 22:45:01.307032 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.307043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.307133 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.307290 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.307291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.307293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:01.307374 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.307376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.308288 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.308289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.309220 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.309222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.309800 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.309801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.309864 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.309870 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.310298 (+ 428us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.310342 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":64,"thread_start_us":73,"threads_started":1}
I20260928 22:45:01.310722 8325 raft_consensus.cc:2454] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabee8b16d274f24a899605a461d5985 in term 0.
I20260928 22:45:01.310886 8186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861351ef1b54c50a79ea9e621a7d325" candidate_uuid: "dabee8b16d274f24a899605a461d5985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8159f2489a24d47a64c90ef74abf334" is_pre_election: true
I20260928 22:45:01.310986 8186 raft_consensus.cc:2454] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabee8b16d274f24a899605a461d5985 in term 0.
I20260928 22:45:01.310995 7980 leader_election.cc:302] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aca0291dfed4b28b5a06fc85ff2c7a9, dabee8b16d274f24a899605a461d5985; no voters:
I20260928 22:45:01.311177 8408 raft_consensus.cc:2781] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:01.311214 8408 raft_consensus.cc:482] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:01.311230 8408 raft_consensus.cc:3037] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.311833 8408 raft_consensus.cc:504] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.311939 8408 leader_election.cc:288] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [CANDIDATE]: Term 1 election: Requested vote from peers f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075), 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577)
I20260928 22:45:01.312081 8186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861351ef1b54c50a79ea9e621a7d325" candidate_uuid: "dabee8b16d274f24a899605a461d5985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8159f2489a24d47a64c90ef74abf334"
I20260928 22:45:01.312101 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861351ef1b54c50a79ea9e621a7d325" candidate_uuid: "dabee8b16d274f24a899605a461d5985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9"
I20260928 22:45:01.312152 8186 raft_consensus.cc:3037] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.312170 8325 raft_consensus.cc:3037] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:01.312677 8186 raft_consensus.cc:2454] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabee8b16d274f24a899605a461d5985 in term 1.
I20260928 22:45:01.312852 7977 leader_election.cc:302] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabee8b16d274f24a899605a461d5985, f8159f2489a24d47a64c90ef74abf334; no voters:
I20260928 22:45:01.312963 8408 raft_consensus.cc:2781] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:01.312990 8325 raft_consensus.cc:2454] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabee8b16d274f24a899605a461d5985 in term 1.
I20260928 22:45:01.313098 8408 raft_consensus.cc:686] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 1 LEADER]: Becoming Leader. State: Replica: dabee8b16d274f24a899605a461d5985, State: Running, Role: LEADER
I20260928 22:45:01.313169 8408 consensus_queue.cc:237] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } }
I20260928 22:45:01.313843 7887 catalog_manager.cc:5495] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 reported cstate change: term changed from 0 to 1, leader changed from <none> to dabee8b16d274f24a899605a461d5985 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dabee8b16d274f24a899605a461d5985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabee8b16d274f24a899605a461d5985" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 } health_report { overall_health: UNKNOWN } } }
I20260928 22:45:01.334435 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.332176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.0.98.1:41785)
0928 22:45:01.332213 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.332214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.332243 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:45:01.332322 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.332323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.332343 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.332407 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.332525 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.333059 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.333746 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.333943 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.333995 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.334129 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.334233 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.334295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.334339 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.334358 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:45:01.339015 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.336947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42543 (local address 127.0.98.2:39075)
0928 22:45:01.336988 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.336990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.336998 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:01.337035 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.337036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.337044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.337084 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.337209 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.337668 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.338377 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.338581 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.338635 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.338783 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.338816 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.338859 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.338908 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.338927 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:45:01.339035 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.336785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39075 (local address 127.0.98.1:42543)
0928 22:45:01.336928 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.336940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.336990 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.337085 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.337086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.337087 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:45:01.337157 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.337158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.337732 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.337733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.338304 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.338306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.338640 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.338641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.338705 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.338718 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.338866 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.338867 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.338906 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.338947 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260928 22:45:01.341110 8186 raft_consensus.cc:1261] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Refusing update from remote peer dabee8b16d274f24a899605a461d5985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.341151 8325 raft_consensus.cc:1261] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Refusing update from remote peer dabee8b16d274f24a899605a461d5985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:01.341356 8477 consensus_queue.cc:1035] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8159f2489a24d47a64c90ef74abf334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.341495 8408 consensus_queue.cc:1035] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aca0291dfed4b28b5a06fc85ff2c7a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:01.348850 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.346149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40967 (local address 127.0.98.1:41785)
0928 22:45:01.346189 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:01.346191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:01.346213 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:45:01.346288 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:01.346289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:01.346299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:01.346350 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:01.346495 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.347058 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.348035 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.348218 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.348277 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.348509 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.348549 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.348601 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:01.348769 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:45:01.348793 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:45:01.348851 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.346119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41785 (local address 127.0.98.2:40967)
0928 22:45:01.346175 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.346185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:01.346227 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:01.346396 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:01.346398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:01.346399 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:45:01.346441 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.346442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.347134 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.347135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.347941 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:01.347944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:01.348290 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:01.348291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:01.348349 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:01.348355 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:45:01.348626 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:45:01.348629 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:01.348686 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:45:01.348772 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:45:01.355685 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.355791 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:01.356424 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1df3bb0268a4e09a9146dffc9c37897: Sending DeleteTablet for 3 replicas of tablet 1861351ef1b54c50a79ea9e621a7d325
I20260928 22:45:01.356668 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260928 22:45:01.356793 8424 tablet_replica.cc:330] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: stopping tablet replica
I20260928 22:45:01.356771 8166 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35628
I20260928 22:45:01.356846 8424 raft_consensus.cc:2229] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.356858 8423 tablet_replica.cc:330] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: stopping tablet replica
I20260928 22:45:01.356876 8424 raft_consensus.cc:2258] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.356897 8423 raft_consensus.cc:2229] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:01.356930 8423 raft_consensus.cc:2258] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.357211 8424 ts_tablet_manager.cc:1905] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.357450 8423 ts_tablet_manager.cc:1905] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.358881 8423 ts_tablet_manager.cc:1918] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.358929 8423 log.cc:1196] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/1861351ef1b54c50a79ea9e621a7d325
I20260928 22:45:01.358937 8026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861351ef1b54c50a79ea9e621a7d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45156
I20260928 22:45:01.359033 8422 tablet_replica.cc:330] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: stopping tablet replica
I20260928 22:45:01.359078 8422 raft_consensus.cc:2229] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:45:01.359123 8423 ts_tablet_manager.cc:1939] T 1861351ef1b54c50a79ea9e621a7d325 P f8159f2489a24d47a64c90ef74abf334: Deleting consensus metadata
I20260928 22:45:01.359155 8422 raft_consensus.cc:2258] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:01.359467 8422 ts_tablet_manager.cc:1905] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:01.360899 8422 ts_tablet_manager.cc:1918] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.360963 8422 log.cc:1196] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/1861351ef1b54c50a79ea9e621a7d325
I20260928 22:45:01.361074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7961
I20260928 22:45:01.361156 8422 ts_tablet_manager.cc:1939] T 1861351ef1b54c50a79ea9e621a7d325 P dabee8b16d274f24a899605a461d5985: Deleting consensus metadata
I20260928 22:45:01.364923 8424 ts_tablet_manager.cc:1918] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:45:01.364992 8424 log.cc:1196] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/1861351ef1b54c50a79ea9e621a7d325
I20260928 22:45:01.365151 8424 ts_tablet_manager.cc:1939] T 1861351ef1b54c50a79ea9e621a7d325 P 4aca0291dfed4b28b5a06fc85ff2c7a9: Deleting consensus metadata
I20260928 22:45:01.365520 7876 catalog_manager.cc:4841] TS 4aca0291dfed4b28b5a06fc85ff2c7a9 (127.0.98.3:33577): tablet 1861351ef1b54c50a79ea9e621a7d325 (table test-table [id=403753469af84d16870ad2f02325ec84]) successfully deleted
I20260928 22:45:01.365636 7873 catalog_manager.cc:4841] TS f8159f2489a24d47a64c90ef74abf334 (127.0.98.2:39075): tablet 1861351ef1b54c50a79ea9e621a7d325 (table test-table [id=403753469af84d16870ad2f02325ec84]) successfully deleted
W20260928 22:45:01.368912 7876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41785 (error 108)
I20260928 22:45:01.369177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8101
I20260928 22:45:01.370754 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:01.370559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41785 (local address 127.0.0.1:45172)
0928 22:45:01.370663 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:01.370710 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
W20260928 22:45:01.370875 7876 catalog_manager.cc:4584] TS dabee8b16d274f24a899605a461d5985 (127.0.98.1:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1861351ef1b54c50a79ea9e621a7d325: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41785: connect: Connection refused (error 111)
I20260928 22:45:01.374562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8241
I20260928 22:45:01.379244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 7857
2026-09-28T22:45:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2296 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[8491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[8491](info): set up 2 sockets
Sep 28 22:45:01 dist-test-slave-pxr1 krb5kdc[8491](info): commencing operation
krb5kdc: starting...
W20260928 22:45:02.325009 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:45:02 dist-test-slave-pxr1 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:45:02.435401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45153
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:02.513206 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:02.513408 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:02.513428 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:02.514550 8504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:02.514583 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:02.514596 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:02.514606 8504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:02.514621 8504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:02.516088 8504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:02.516284 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:02.516460 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:02.521060 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:02.521114 8504 server_base.cc:1034] running on GCE node
W20260928 22:45:02.521067 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:02.521067 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:02.521452 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:02.521705 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:02.522837 8504 hybrid_clock.cc:648] HybridClock initialized: now 1790635502522825 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T22:45:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1154 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:45:02 dist-test-slave-pxr1 krb5kdc[8517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:45:02 dist-test-slave-pxr1 krb5kdc[8517](info): set up 2 sockets
Sep 28 22:45:02 dist-test-slave-pxr1 krb5kdc[8517](info): commencing operation
krb5kdc: starting...
W20260928 22:45:03.484912 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:45:03 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:45:03.595707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:03.674170 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:03.674393 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:03.674425 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:03.675635 8530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:03.675684 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:03.675701 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:03.675719 8530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:03.675732 8530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:03.677309 8530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:03.677565 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:03.677784 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:03.682495 8538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:03.682495 8536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:03.682490 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:03.682679 8530 server_base.cc:1034] running on GCE node
I20260928 22:45:03.682976 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:03.683279 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:03.684419 8530 hybrid_clock.cc:648] HybridClock initialized: now 1790635503684406 us; error 30 us; skew 500 ppm
Sep 28 22:45:03 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:03.686802 8530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:45:03.687208 8530 webserver.cc:460] Webserver started at http://127.0.98.62:41573/ using document root <none> and password file <none>
I20260928 22:45:03.687430 8530 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:03.687476 8530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:03.687599 8530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:03.688634 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "559eaca0ce684d1a93c47c72136025c1"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "07ef7b0a175074e45a7923ee261f36ed"
server_key_iv: "214d106f8ac1d3aef083ab6e901e0564"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.689009 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "559eaca0ce684d1a93c47c72136025c1"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "07ef7b0a175074e45a7923ee261f36ed"
server_key_iv: "214d106f8ac1d3aef083ab6e901e0564"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.690282 8530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:03.690956 8545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:03.691108 8530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:03.691197 8530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "559eaca0ce684d1a93c47c72136025c1"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "07ef7b0a175074e45a7923ee261f36ed"
server_key_iv: "214d106f8ac1d3aef083ab6e901e0564"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.691258 8530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:03.714538 8530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:03.716693 8530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:03.716871 8530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:03.720506 8530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35459
I20260928 22:45:03.720558 8597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35459 every 8 connection(s)
I20260928 22:45:03.721048 8530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:03.721709 8598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:03.724025 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Bootstrap starting.
I20260928 22:45:03.724606 8598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:03.724901 8598 log.cc:824] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:03.725550 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: No bootstrap required, opened a new log
I20260928 22:45:03.727149 8598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:03.727299 8598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:03.727320 8598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559eaca0ce684d1a93c47c72136025c1, State: Initialized, Role: FOLLOWER
I20260928 22:45:03.727451 8598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:03.727522 8598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:03.727545 8598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:03.727577 8598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:03.728475 8598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:03.728574 8598 leader_election.cc:302] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559eaca0ce684d1a93c47c72136025c1; no voters:
I20260928 22:45:03.728765 8598 leader_election.cc:288] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:45:03.728803 8601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:03.728942 8601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 LEADER]: Becoming Leader. State: Replica: 559eaca0ce684d1a93c47c72136025c1, State: Running, Role: LEADER
I20260928 22:45:03.729043 8598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:03.729029 8601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:03.729423 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "559eaca0ce684d1a93c47c72136025c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260928 22:45:03.729449 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559eaca0ce684d1a93c47c72136025c1. Latest consensus state: current_term: 1 leader_uuid: "559eaca0ce684d1a93c47c72136025c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260928 22:45:03.729578 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:03.729601 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:03.730052 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:03.730296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8530
I20260928 22:45:03.730408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:45:03.730536 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:45:03.730610 392 external_mini_cluster.cc:1428] Setting key 2dc551203d7a5ece705309c40c351cc7
I20260928 22:45:03.732472 8611 catalog_manager.cc:1347] Generated new cluster ID: c51e24d5f5fd4c7684bda94369626577
I20260928 22:45:03.732523 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:03.744436 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:45:03.745296 8611 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 22:45:03 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:03.750514 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Generated new TSK 0
I20260928 22:45:03.750691 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:45:03.751662 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:03.731211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.98.62:35459)
0928 22:45:03.731386 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:03.731395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:03.731428 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:45:03.731544 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:03.731551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:03.731602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:03.731727 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:03.740921 (+ 9194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.742450 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.743565 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.744261 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.744381 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:03.747621 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:03.747627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:03.747627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:03.747642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:03.750047 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:03.750447 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:03.750451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:03.750452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:03.750579 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:03.750803 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:03.750804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:03.750805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:03.751061 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:03.751180 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:03.751320 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 22:45:03.751443 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:45:03.775920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:03.855983 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:03.856194 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:03.856217 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:03.857422 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:03.857468 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:03.857496 8624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:03.859002 8624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:03.859222 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:03.859444 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:03.864116 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:03.864107 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:03.864107 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:03.864320 8624 server_base.cc:1034] running on GCE node
I20260928 22:45:03.864570 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:03.864820 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:03.865968 8624 hybrid_clock.cc:648] HybridClock initialized: now 1790635503865953 us; error 31 us; skew 500 ppm
Sep 28 22:45:03 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:03.868369 8624 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:45:03.868824 8624 webserver.cc:460] Webserver started at http://127.0.98.1:42267/ using document root <none> and password file <none>
I20260928 22:45:03.869015 8624 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:03.869062 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:03.869191 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:03.870219 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "93a37e282c434ef48b9694de2892deb6"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "9fadee753ed4f2b504003426dd4ed35f"
server_key_iv: "b20e82c6d420638b00331d65649b2da2"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.870563 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93a37e282c434ef48b9694de2892deb6"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "9fadee753ed4f2b504003426dd4ed35f"
server_key_iv: "b20e82c6d420638b00331d65649b2da2"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.871806 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:45:03.872504 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:03.872689 8624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:03.872797 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "93a37e282c434ef48b9694de2892deb6"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "9fadee753ed4f2b504003426dd4ed35f"
server_key_iv: "b20e82c6d420638b00331d65649b2da2"
server_key_version: "encryptionkey@0"
I20260928 22:45:03.872864 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:03.886267 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:03.888219 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:03.888355 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:03.888571 8624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:03.888932 8624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:03.888967 8624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:03.888998 8624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:03.889019 8624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:03.894475 8624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38627
I20260928 22:45:03.894536 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38627 every 8 connection(s)
I20260928 22:45:03.894821 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:03.900866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8624
I20260928 22:45:03.900972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
Sep 28 22:45:03 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:03.901181 392 external_mini_cluster.cc:1428] Setting key b587c45f14fed89f2e2a1e0cf764f975
I20260928 22:45:03.904281 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:03.895549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60799 (local address 127.0.98.62:35459)
0928 22:45:03.895607 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:03.895609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:03.895837 (+ 228us) server_negotiation.cc:407] Connection header received
0928 22:45:03.896770 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:03.896771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:03.896782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:03.896848 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:03.897478 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.897993 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.899083 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.899291 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.899355 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:03.901549 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:03.901551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:03.901551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:03.901560 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:03.902958 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:03.903347 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:03.903348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:03.903349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:03.903414 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:03.903637 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:03.903638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:03.903638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:03.903755 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:03.903813 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:03.904081 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 22:45:03.904178 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:45:03.904428 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:03.895533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:60799)
0928 22:45:03.895690 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:03.895733 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:03.896630 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:03.896910 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:03.896914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:03.897175 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:03.897370 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:03.897374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.898140 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.898141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:03.898965 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:03.898967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:03.899367 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:03.899368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:03.899471 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:03.899948 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:03.899956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:03.901456 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:03.903033 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:03.903040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:03.903043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:03.903268 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:03.903471 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:03.903471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:03.903472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:03.903579 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:03.903810 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:03.903817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:03.903959 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 22:45:03.904047 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 22:45:03.904208 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 22:45:03.904899 8753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35459
I20260928 22:45:03.905012 8753 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:03.905222 8753 heartbeater.cc:511] Master 127.0.98.62:35459 requested a full tablet report, sending...
I20260928 22:45:03.905730 8562 ts_manager.cc:194] Registered new tserver with Master: 93a37e282c434ef48b9694de2892deb6 (127.0.98.1:38627)
I20260928 22:45:03.906338 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60799
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:45:03.928067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:04.008831 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.009043 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.009068 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.010234 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.010274 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.010300 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:04.011793 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.011982 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.012166 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.016940 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.016947 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.017030 8758 server_base.cc:1034] running on GCE node
W20260928 22:45:04.017012 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.017390 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.017613 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.018765 8758 hybrid_clock.cc:648] HybridClock initialized: now 1790635504018748 us; error 33 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.021263 8758 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:45:04.021703 8758 webserver.cc:460] Webserver started at http://127.0.98.2:41939/ using document root <none> and password file <none>
I20260928 22:45:04.021906 8758 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.021953 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.022076 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:04.023087 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e68c682fcb34a71a559658c88b0fe2a"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "7f3663de5a21a39b6029492c22c23c43"
server_key_iv: "0b0ea64b2fadcceab467c2447bd14e6a"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.023445 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e68c682fcb34a71a559658c88b0fe2a"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "7f3663de5a21a39b6029492c22c23c43"
server_key_iv: "0b0ea64b2fadcceab467c2447bd14e6a"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.024657 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:45:04.025354 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.025506 8758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:04.025586 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "9e68c682fcb34a71a559658c88b0fe2a"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "7f3663de5a21a39b6029492c22c23c43"
server_key_iv: "0b0ea64b2fadcceab467c2447bd14e6a"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.025653 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.041529 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.043431 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.043561 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.043776 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:04.044056 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:04.044088 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.044126 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:04.044152 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.049558 8758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32925
I20260928 22:45:04.049609 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32925 every 8 connection(s)
I20260928 22:45:04.049912 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:45:04.053692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8758
I20260928 22:45:04.053802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:45:04.053978 392 external_mini_cluster.cc:1428] Setting key 551c49f4700b89b14a03630608e81669
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:04.059587 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.050615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43617 (local address 127.0.98.62:35459)
0928 22:45:04.050666 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.050667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.050873 (+ 206us) server_negotiation.cc:407] Connection header received
0928 22:45:04.051837 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.051839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.051855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.051909 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.052564 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.053108 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.054239 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.054500 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.054539 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.056912 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:04.056913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:04.056914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:04.056922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:04.058329 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.058687 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.058689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.058689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.058749 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.058950 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.058951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.058951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.059088 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:04.059165 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:04.059404 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 22:45:04.059492 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:45:04.059645 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.050585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:43617)
0928 22:45:04.050757 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.050793 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.051726 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.051983 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.051987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.052250 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.052475 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.052480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.053193 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.053195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.054130 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.054133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.054629 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.054630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.054790 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.055406 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.055414 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.056816 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:04.058401 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:04.058407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:04.058409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:04.058620 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:04.058809 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:04.058810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:04.058810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:04.058901 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:04.059164 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:04.059168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:04.059286 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 22:45:04.059366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:45:04.059484 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260928 22:45:04.060074 8887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35459
I20260928 22:45:04.060163 8887 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:04.060334 8887 heartbeater.cc:511] Master 127.0.98.62:35459 requested a full tablet report, sending...
I20260928 22:45:04.060707 8562 ts_manager.cc:194] Registered new tserver with Master: 9e68c682fcb34a71a559658c88b0fe2a (127.0.98.2:32925)
I20260928 22:45:04.061097 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43617
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:45:04.078837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:04.157606 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.157820 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.157843 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.159009 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.159060 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.159099 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:04.160642 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.160872 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.161059 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.165746 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.165755 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.165759 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.166054 8892 server_base.cc:1034] running on GCE node
I20260928 22:45:04.166244 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.166474 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.167618 8892 hybrid_clock.cc:648] HybridClock initialized: now 1790635504167597 us; error 34 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.170025 8892 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:45:04.170428 8892 webserver.cc:460] Webserver started at http://127.0.98.3:44051/ using document root <none> and password file <none>
I20260928 22:45:04.170599 8892 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.170636 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.170737 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:04.171739 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "a7249fa1ba514b3ba4668a3570fe1d89"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "dbd0a4f3d424432280a9bd8e509895de"
server_key_iv: "43e533435981e65be914455450c7b757"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.172029 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a7249fa1ba514b3ba4668a3570fe1d89"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "dbd0a4f3d424432280a9bd8e509895de"
server_key_iv: "43e533435981e65be914455450c7b757"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.173238 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:45:04.173915 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.174048 8892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:04.174117 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "a7249fa1ba514b3ba4668a3570fe1d89"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "dbd0a4f3d424432280a9bd8e509895de"
server_key_iv: "43e533435981e65be914455450c7b757"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.174167 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.194478 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.196223 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.196331 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.196511 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:04.196821 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:04.196858 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.196887 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:04.196911 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.202167 8892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36437
I20260928 22:45:04.202211 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36437 every 8 connection(s)
I20260928 22:45:04.202508 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:45:04.203445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 8892
I20260928 22:45:04.203559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:45:04.203724 392 external_mini_cluster.cc:1428] Setting key f1fa8ed9fe0e6908aa8397a47ab2bff4
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:04.211045 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.203168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46637 (local address 127.0.98.62:35459)
0928 22:45:04.203220 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.203222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.203508 (+ 286us) server_negotiation.cc:407] Connection header received
0928 22:45:04.204566 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.204567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.204578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.204618 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.205294 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.205783 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.206693 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.206872 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.206914 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.208833 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:04.208834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:04.208835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:04.208841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:04.210050 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.210368 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.210368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.210369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.210414 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.210584 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.210584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.210584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.210680 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:04.210723 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:04.210920 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 22:45:04.210978 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:45:04.211234 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.203148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:46637)
0928 22:45:04.203359 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.203409 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.204490 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.204680 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.204683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.205016 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.205226 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.205231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.205843 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.205844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.206634 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.206637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.206923 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.206924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.207016 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.207450 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.207457 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.208708 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:04.210109 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:04.210112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:04.210114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:04.210316 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:04.210456 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:04.210457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:04.210457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:04.210543 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:04.210720 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:04.210723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:04.210832 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 22:45:04.210907 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:45:04.211019 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260928 22:45:04.211655 9021 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35459
I20260928 22:45:04.211747 9021 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:04.211917 9021 heartbeater.cc:511] Master 127.0.98.62:35459 requested a full tablet report, sending...
I20260928 22:45:04.212265 8562 ts_manager.cc:194] Registered new tserver with Master: a7249fa1ba514b3ba4668a3570fe1d89 (127.0.98.3:36437)
I20260928 22:45:04.212597 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46637
I20260928 22:45:04.215291 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 22:45:04.281297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8624
I20260928 22:45:04.285251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8758
I20260928 22:45:04.289198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8892
I20260928 22:45:04.292804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 8530
I20260928 22:45:04.297031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--webserver_interface=127.0.98.62
--webserver_port=41573
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:04.376605 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.376868 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.376893 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.378123 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:04.378168 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.378181 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.378193 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:04.378203 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:04.379750 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.379983 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.380198 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.384756 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.384756 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.384871 9030 server_base.cc:1034] running on GCE node
W20260928 22:45:04.384794 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.385217 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.385486 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.386627 9030 hybrid_clock.cc:648] HybridClock initialized: now 1790635504386609 us; error 33 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.388942 9030 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:45:04.389370 9030 webserver.cc:460] Webserver started at http://127.0.98.62:41573/ using document root <none> and password file <none>
I20260928 22:45:04.389572 9030 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.389633 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.390904 9030 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:04.391464 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.391633 9030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:04.391716 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "559eaca0ce684d1a93c47c72136025c1"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "07ef7b0a175074e45a7923ee261f36ed"
server_key_iv: "214d106f8ac1d3aef083ab6e901e0564"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.392048 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.400101 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.402125 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.402262 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.406015 9030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35459
I20260928 22:45:04.406041 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35459 every 8 connection(s)
I20260928 22:45:04.406538 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:04.407733 9098 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:45:04.408332 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Bootstrap starting.
I20260928 22:45:04.411123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9030
I20260928 22:45:04.411378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38627
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35459
--webserver_port=42267
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:45:04.416419 9098 log.cc:824] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:04.418062 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:45:04.418309 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Bootstrap complete.
I20260928 22:45:04.420274 9098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:04.420491 9098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559eaca0ce684d1a93c47c72136025c1, State: Initialized, Role: FOLLOWER
I20260928 22:45:04.420634 9098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:04.420706 9098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:04.420773 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:04.420819 9098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:45:04.421770 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:04.421861 9098 leader_election.cc:302] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559eaca0ce684d1a93c47c72136025c1; no voters:
I20260928 22:45:04.422024 9098 leader_election.cc:288] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:45:04.422067 9102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:45:04.422259 9102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [term 2 LEADER]: Becoming Leader. State: Replica: 559eaca0ce684d1a93c47c72136025c1, State: Running, Role: LEADER
I20260928 22:45:04.422288 9098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:04.422485 9102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260928 22:45:04.422806 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "559eaca0ce684d1a93c47c72136025c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260928 22:45:04.422880 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:04.423254 9109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:04.423785 9109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:45:04.423980 9109 catalog_manager.cc:1259] Loaded cluster ID: c51e24d5f5fd4c7684bda94369626577
I20260928 22:45:04.424104 9109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:04.424966 9109 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:45:04.425140 9109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1: Loaded TSK: 0
I20260928 22:45:04.425333 9109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:45:04.425436 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559eaca0ce684d1a93c47c72136025c1. Latest consensus state: current_term: 2 leader_uuid: "559eaca0ce684d1a93c47c72136025c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559eaca0ce684d1a93c47c72136025c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260928 22:45:04.425496 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559eaca0ce684d1a93c47c72136025c1 [sys.catalog]: This master's current role is: LEADER
W20260928 22:45:04.497233 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.497437 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.497469 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.498637 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.498680 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.498716 9100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:04.500176 9100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42267
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.500406 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.500620 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.506382 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.506490 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.506382 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.506659 9100 server_base.cc:1034] running on GCE node
I20260928 22:45:04.506878 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.507114 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.508270 9100 hybrid_clock.cc:648] HybridClock initialized: now 1790635504508255 us; error 34 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.510723 9100 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 22:45:04.511185 9100 webserver.cc:460] Webserver started at http://127.0.98.1:42267/ using document root <none> and password file <none>
I20260928 22:45:04.511402 9100 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.511456 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.512836 9100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:04.513437 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.513612 9100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:04.513695 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "93a37e282c434ef48b9694de2892deb6"
format_stamp: "Formatted at 2026-09-28 22:45:03 on dist-test-slave-pxr1"
server_key: "9fadee753ed4f2b504003426dd4ed35f"
server_key_iv: "b20e82c6d420638b00331d65649b2da2"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.514040 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.524156 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.526243 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.526372 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.526571 9100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:04.526837 9100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:04.526868 9100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.526896 9100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:04.526921 9100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.532040 9100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38627
I20260928 22:45:04.532326 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38627 every 8 connection(s)
I20260928 22:45:04.532471 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:04.536494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9100
I20260928 22:45:04.536842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32925
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35459
--webserver_port=41939
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.544265 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.533256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:43477)
0928 22:45:04.533435 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.533475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.533924 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.537244 (+ 3320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.537248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.537837 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.538072 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.538079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.539722 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.539724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.540562 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.540565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.541308 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.541310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.541436 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.541954 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.541962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.544049 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
W20260928 22:45:04.544571 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.533334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43477 (local address 127.0.98.62:35459)
0928 22:45:04.536760 (+ 3426us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.536770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.536813 (+ 43us) server_negotiation.cc:407] Connection header received
0928 22:45:04.536946 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.536954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.537016 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.537207 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.538205 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.539609 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.540652 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.541221 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.541324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.544200 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43477: BlockingRecv error: recv got EOF from 127.0.98.1:43477 (error 108)
Metrics: {"server-negotiator.queue_time_us":3337,"thread_start_us":86,"threads_started":1}
W20260928 22:45:04.544574 9248 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35459 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35459: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.548830 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.544891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:55321)
0928 22:45:04.544924 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.544935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.544976 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.545122 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.545123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.545242 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.545285 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.545287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.545900 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.545901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.546399 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.546401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.546728 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.546729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.546789 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.547188 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.547193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.548760 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 22:45:04.548885 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.544945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55321 (local address 127.0.98.62:35459)
0928 22:45:04.544975 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.544977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.544985 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:04.545044 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.545045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.545062 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.545109 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.545350 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.545832 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.546472 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.546673 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.546726 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.548836 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55321: BlockingRecv error: recv got EOF from 127.0.98.1:55321 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.553339 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.549398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:42737)
0928 22:45:04.549485 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.549495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.549540 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.549656 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.549657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.549772 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.549800 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.549801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.550384 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.550385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.550914 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.550916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.551296 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.551297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.551341 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.551718 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.551724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.553271 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 22:45:04.553409 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.549450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42737 (local address 127.0.98.62:35459)
0928 22:45:04.549491 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.549493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.549525 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:45:04.549600 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.549601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.549611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.549654 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.549856 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.550317 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.550985 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.551187 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.551246 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.553357 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42737: BlockingRecv error: recv got EOF from 127.0.98.1:42737 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:45:04.553671 9248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:45:04.623895 9252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.624092 9252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.624116 9252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.625305 9252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.625340 9252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.625365 9252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:04.626719 9252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41939
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.626904 9252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.627084 9252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.631764 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.631765 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.631770 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.631791 9252 server_base.cc:1034] running on GCE node
I20260928 22:45:04.632144 9252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.632390 9252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.633519 9252 hybrid_clock.cc:648] HybridClock initialized: now 1790635504633507 us; error 29 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.635778 9252 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 22:45:04.636195 9252 webserver.cc:460] Webserver started at http://127.0.98.2:41939/ using document root <none> and password file <none>
I20260928 22:45:04.636389 9252 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.636446 9252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.637755 9252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:04.638243 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.638397 9252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.638475 9252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "9e68c682fcb34a71a559658c88b0fe2a"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "7f3663de5a21a39b6029492c22c23c43"
server_key_iv: "0b0ea64b2fadcceab467c2447bd14e6a"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.638803 9252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.649616 9252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.651314 9252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.651449 9252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.651683 9252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:04.652027 9252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:04.652069 9252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.652122 9252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:04.652148 9252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.657789 9252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32925
I20260928 22:45:04.657828 9380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32925 every 8 connection(s)
I20260928 22:45:04.658154 9252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:45:04.662729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9252
I20260928 22:45:04.663026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36437
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35459
--webserver_port=44051
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.665205 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.658842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:58855)
0928 22:45:04.658994 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.659030 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.659495 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.659687 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.659690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.660243 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.660436 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.660440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.661057 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.661058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.661883 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.661886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.662241 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.662242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.662357 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.662832 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.662844 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.665059 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260928 22:45:04.665212 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.658843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58855 (local address 127.0.98.62:35459)
0928 22:45:04.658888 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.658891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.659112 (+ 221us) server_negotiation.cc:407] Connection header received
0928 22:45:04.659568 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.659569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.659582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.659631 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.660503 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.660999 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.661979 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.662162 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.662220 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.665155 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58855: BlockingRecv error: recv got EOF from 127.0.98.2:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:45:04.665413 9381 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35459 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35459: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.669320 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.665621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:58411)
0928 22:45:04.665672 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.665680 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.665713 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.665817 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.665818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.665933 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.665977 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.665978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.666559 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.666560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.667060 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.667061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.667368 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.667369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.667428 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.667794 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.667798 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.669258 (+ 1460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:45:04.669402 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.665677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58411 (local address 127.0.98.62:35459)
0928 22:45:04.665727 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.665729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.665736 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:04.665762 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.665763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.665771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.665816 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.666036 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.666496 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.667128 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.667312 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.667365 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.669346 (+ 1981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58411: BlockingRecv error: recv got EOF from 127.0.98.2:58411 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.673475 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.669687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:45971)
0928 22:45:04.669728 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.669736 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.669765 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.669897 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.669897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.670008 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.670036 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.670037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.670619 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.670620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.671114 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.671115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.671425 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.671425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.671474 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.671841 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.671846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.673426 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:45:04.673585 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.669755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45971 (local address 127.0.98.62:35459)
0928 22:45:04.669804 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.669806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.669813 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:04.669833 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.669833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.669842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.669887 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.670092 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.670554 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.671187 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.671371 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.671422 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.673527 (+ 2105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45971: BlockingRecv error: recv got EOF from 127.0.98.2:45971 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 22:45:04.673585 9381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:45:04.743302 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:04.743525 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:04.743551 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:04.744750 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:04.744787 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:04.744813 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:04.746277 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44051
--tserver_master_addrs=127.0.98.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:04.746480 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:04.746661 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:04.751530 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:04.751537 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.751616 9384 server_base.cc:1034] running on GCE node
W20260928 22:45:04.751545 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:04.751950 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:04.752206 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:04.753340 9384 hybrid_clock.cc:648] HybridClock initialized: now 1790635504753329 us; error 30 us; skew 500 ppm
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635504, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:04.755632 9384 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 22:45:04.756043 9384 webserver.cc:460] Webserver started at http://127.0.98.3:44051/ using document root <none> and password file <none>
I20260928 22:45:04.756265 9384 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:04.756328 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:04.757622 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:04.758144 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.758296 9384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:04.758373 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "a7249fa1ba514b3ba4668a3570fe1d89"
format_stamp: "Formatted at 2026-09-28 22:45:04 on dist-test-slave-pxr1"
server_key: "dbd0a4f3d424432280a9bd8e509895de"
server_key_iv: "43e533435981e65be914455450c7b757"
server_key_version: "encryptionkey@0"
I20260928 22:45:04.758710 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:04.773751 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:04.775472 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:04.775606 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:04.775815 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:04.776098 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:04.776129 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.776160 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:04.776191 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:04.781646 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36437
I20260928 22:45:04.781682 9512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36437 every 8 connection(s)
I20260928 22:45:04.782011 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.787875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9384
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:45:04.789124 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.782687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55665 (local address 127.0.98.62:35459)
0928 22:45:04.782732 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.782733 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.782982 (+ 249us) server_negotiation.cc:407] Connection header received
0928 22:45:04.783393 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.783394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.783403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.783454 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.784359 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.784902 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.785900 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.786092 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.786158 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.789066 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55665: BlockingRecv error: recv got EOF from 127.0.98.3:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:45:04.789194 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.782673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:55665)
0928 22:45:04.782813 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.782849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.783313 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.783514 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.783518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.784084 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.784286 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.784291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.784965 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.784966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.785799 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.785802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.786167 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.786168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.786276 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.786774 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.786782 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.788989 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260928 22:45:04.789422 9513 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35459 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35459: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.793707 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.789635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:50011)
0928 22:45:04.789679 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.789689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.789725 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.789844 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.789845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.789971 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.790024 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.790026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.790580 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.790581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.791223 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.791226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.791608 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.791609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.791670 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.791964 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.791968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.793639 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:45:04.793787 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.789683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50011 (local address 127.0.98.62:35459)
0928 22:45:04.789716 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.789718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.789725 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:04.789773 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.789773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.789782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.789836 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.790090 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.790515 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.791294 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.791544 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.791592 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.793734 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50011: BlockingRecv error: recv got EOF from 127.0.98.3:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:45:04.795296 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.788378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.0.98.62:35459)
0928 22:45:04.788494 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.788496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.788504 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:04.788568 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.788569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.788582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.788634 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.789247 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.789956 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.790773 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.790957 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.791004 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.792362 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:04.792366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:04.792366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:04.792376 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:04.794037 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.794353 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.794356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.794357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.794455 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:04.794624 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:04.794625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:04.794625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:04.794855 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:04.794979 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:04.795094 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:45:04.795206 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:04 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:04.797816 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:04.793956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:54053)
0928 22:45:04.794118 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:04.794128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:04.794171 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:04.794291 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:04.794292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:04.794409 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:04.794435 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.794437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.795066 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.795067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.795756 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:04.795759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.796054 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.796055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:04.796102 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.796425 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:04.796429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:04.797753 (+ 1324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
W20260928 22:45:04.797879 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:04.794077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54053 (local address 127.0.98.62:35459)
0928 22:45:04.794112 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:04.794113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:04.794158 (+ 45us) server_negotiation.cc:407] Connection header received
0928 22:45:04.794230 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:04.794231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:04.794239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:04.794283 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:04.794493 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.795000 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.795824 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:04.796001 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:04.796050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:04.797834 (+ 1784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54053: BlockingRecv error: recv got EOF from 127.0.98.3:54053 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 22:45:04.797937 9513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:05.558261 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:05.554011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:41751)
0928 22:45:05.554128 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:05.554139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:05.554188 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:05.554331 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:05.554331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:05.554477 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:05.554523 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.554524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.555118 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.555118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.555695 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.555697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.556110 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.556112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.556178 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.556585 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:05.556590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:05.558197 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
W20260928 22:45:05.558349 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:05.554100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41751 (local address 127.0.98.62:35459)
0928 22:45:05.554216 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:05.554218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:05.554225 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:05.554247 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:05.554248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:05.554257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:05.554321 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:05.554577 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.555062 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.555778 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.556033 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.556098 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.558284 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41751: BlockingRecv error: recv got EOF from 127.0.98.1:41751 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:05.678133 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:05.673905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:42865)
0928 22:45:05.674027 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:05.674039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:05.674129 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:05.674263 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:05.674263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:05.674430 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:05.674478 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.674480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.675120 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.675122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.675664 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.675665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.675989 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.675990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.676041 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.676468 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:05.676471 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:05.678037 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
W20260928 22:45:05.678196 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:05.673982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42865 (local address 127.0.98.62:35459)
0928 22:45:05.674046 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:05.674048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:05.674088 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:45:05.674188 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:05.674189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:05.674201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:05.674247 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:05.674545 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.675036 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.675744 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.675930 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.675984 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.678144 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42865: BlockingRecv error: recv got EOF from 127.0.98.2:42865 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:05 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:05.802959 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:05.798345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:36587)
0928 22:45:05.798480 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:05.798491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:05.798553 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:05.798683 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:05.798684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:05.798835 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:05.798883 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.798884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.799499 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.799500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.800067 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:05.800069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.800426 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.800428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:05.800478 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.801023 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:05.801028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:05.802890 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
W20260928 22:45:05.803066 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:05.798417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36587 (local address 127.0.98.62:35459)
0928 22:45:05.798469 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:05.798471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:05.798550 (+ 79us) server_negotiation.cc:407] Connection header received
0928 22:45:05.798598 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:05.798598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:05.798612 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:05.798671 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:05.798938 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.799430 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.800155 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:05.800340 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:05.800388 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:05.803004 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36587: BlockingRecv error: recv got EOF from 127.0.98.3:36587 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:06.562968 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:06.558713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:38711)
0928 22:45:06.558856 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:06.558869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:06.558943 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:06.559070 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:06.559070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:06.559225 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:06.559271 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.559272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.559884 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.559885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.560421 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.560423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.560747 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.560748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.560791 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.561262 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:06.561266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:06.562909 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
W20260928 22:45:06.563094 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:06.558791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38711 (local address 127.0.98.62:35459)
0928 22:45:06.558924 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:06.558925 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:06.558939 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:45:06.558988 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:06.558989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:06.558999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:06.559054 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:06.559325 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.559814 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.560485 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.560671 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.560745 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.563016 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38711: BlockingRecv error: recv got EOF from 127.0.98.1:38711 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:06.682879 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:06.678644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:55661)
0928 22:45:06.678757 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:06.678768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:06.678834 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:06.678954 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:06.678955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:06.679144 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:06.679189 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.679191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.679845 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.679846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.680383 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.680384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.680756 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.680757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.680815 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.681252 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:06.681256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:06.682805 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
W20260928 22:45:06.682950 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:06.678745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55661 (local address 127.0.98.62:35459)
0928 22:45:06.678823 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:06.678825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:06.678834 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:06.678890 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:06.678891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:06.678902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:06.678949 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:06.679265 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.679769 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.680460 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.680661 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.680738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.682898 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55661: BlockingRecv error: recv got EOF from 127.0.98.2:55661 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:06 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:06.807577 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:06.803392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:33129)
0928 22:45:06.803533 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:06.803544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:06.803599 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:06.803724 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:06.803724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:06.803876 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:06.803924 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.803926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.804565 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.804565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.805155 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:06.805158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.805500 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.805501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:06.805538 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.805955 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:06.805959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:06.807516 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
W20260928 22:45:06.807694 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:06.803515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33129 (local address 127.0.98.62:35459)
0928 22:45:06.803566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:06.803568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:06.803587 (+ 19us) server_negotiation.cc:407] Connection header received
0928 22:45:06.803643 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:06.803644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:06.803654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:06.803703 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:06.803989 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.804503 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.805250 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:06.805439 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:06.805498 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:06.807633 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33129: BlockingRecv error: recv got EOF from 127.0.98.3:33129 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:07.571059 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:07.563385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:52435)
0928 22:45:07.563534 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:07.563544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:07.563594 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:07.563755 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:07.563756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:07.563908 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:07.563957 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.563959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.564569 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.564569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.565165 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.565169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.565508 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.565509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.565552 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.569340 (+ 3788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:07.569346 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:07.570984 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260928 22:45:07.571156 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:07.563499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52435 (local address 127.0.98.62:35459)
0928 22:45:07.563627 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:07.563629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:07.563636 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:07.563660 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:07.563661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:07.563671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:07.563740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:07.564015 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.564511 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.565248 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.565448 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.565504 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.571088 (+ 5584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52435: BlockingRecv error: recv got EOF from 127.0.98.1:52435 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:07.688030 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:07.683527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:35185)
0928 22:45:07.683681 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:07.683691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:07.683751 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:07.683872 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:07.683872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:07.684025 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:07.684072 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.684074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.684804 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.684805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.685516 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.685519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.685883 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.685885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.685935 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.686349 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:07.686353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:07.687958 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260928 22:45:07.688114 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:07.683603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35185 (local address 127.0.98.62:35459)
0928 22:45:07.683660 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:07.683662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:07.683746 (+ 84us) server_negotiation.cc:407] Connection header received
0928 22:45:07.683795 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:07.683795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:07.683806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:07.683859 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:07.684118 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.684683 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.685590 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.685785 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.685835 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.688054 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35185: BlockingRecv error: recv got EOF from 127.0.98.2:35185 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:07 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:07.812217 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:07.808045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:50823)
0928 22:45:07.808183 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:07.808194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:07.808262 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:07.808385 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:07.808385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:07.808552 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:07.808599 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.808601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.809205 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.809206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.809731 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:07.809732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.810019 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.810020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:07.810056 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.810471 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:07.810475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:07.812152 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
W20260928 22:45:07.812305 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:07.808124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50823 (local address 127.0.98.62:35459)
0928 22:45:07.808213 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:07.808215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:07.808245 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:45:07.808311 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:07.808312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:07.808323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:07.808370 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:07.808649 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.809155 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.809792 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:07.809969 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:07.810011 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:07.812246 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50823: BlockingRecv error: recv got EOF from 127.0.98.3:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:08.575897 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:08.571516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:45593)
0928 22:45:08.571630 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:08.571641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:08.571693 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:08.571822 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:08.571823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:08.571976 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:08.572026 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.572028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.572677 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.572678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.573286 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.573289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.573657 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.573657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.573702 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.574143 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:08.574148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:08.575818 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260928 22:45:08.576020 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:08.571570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45593 (local address 127.0.98.62:35459)
0928 22:45:08.571709 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:08.571710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:08.571719 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:08.571740 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:08.571740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:08.571750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:08.571807 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:08.572085 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.572592 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.573376 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.573589 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.573642 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.575953 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45593: BlockingRecv error: recv got EOF from 127.0.98.1:45593 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:08.693032 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:08.688547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:35347)
0928 22:45:08.688674 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:08.688684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:08.688808 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:08.688934 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:08.688935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:08.689088 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:08.689134 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.689136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.689823 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.689823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.690469 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.690472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.690825 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.690826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.690878 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.691303 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:08.691308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:08.692972 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
W20260928 22:45:08.693120 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:08.688652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35347 (local address 127.0.98.62:35459)
0928 22:45:08.688699 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:08.688701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:08.688791 (+ 90us) server_negotiation.cc:407] Connection header received
0928 22:45:08.688856 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:08.688856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:08.688867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:08.688916 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:08.689186 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.689724 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.690543 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.690728 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.690780 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.693061 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35347: BlockingRecv error: recv got EOF from 127.0.98.2:35347 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:08 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:08.817193 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:08.812835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:57981)
0928 22:45:08.812966 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:08.812978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:08.813042 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:08.813192 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:08.813193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:08.813356 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:08.813409 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.813411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.814059 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.814060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.814575 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:08.814577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.814923 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.814924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:08.814977 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.815421 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:08.815425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:08.817115 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260928 22:45:08.817271 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:08.812969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57981 (local address 127.0.98.62:35459)
0928 22:45:08.813065 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:08.813066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:08.813076 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:08.813100 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:08.813101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:08.813113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:08.813160 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:08.813495 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.813984 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.814659 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:08.814856 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:08.814909 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:08.817217 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57981: BlockingRecv error: recv got EOF from 127.0.98.3:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:09.580888 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:09.576475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:36215)
0928 22:45:09.576636 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:09.576650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:09.576741 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:09.576912 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:09.576913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:09.577081 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:09.577129 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.577130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.577778 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.577779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.578317 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.578319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.578615 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.578616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.578652 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.579063 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:09.579068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:09.580827 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
W20260928 22:45:09.581001 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:09.576574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36215 (local address 127.0.98.62:35459)
0928 22:45:09.576777 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:09.576779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:09.576790 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:09.576817 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:09.576818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:09.576828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:09.576888 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:09.577196 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.577698 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.578379 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.578565 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.578608 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.580933 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36215: BlockingRecv error: recv got EOF from 127.0.98.1:36215 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:09.697672 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:09.693551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:35981)
0928 22:45:09.693702 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:09.693716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:09.693774 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:09.693907 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:09.693907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:09.694061 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:09.694109 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.694110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.694719 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.694720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.695277 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.695279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.695587 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.695588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.695626 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.696027 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:09.696030 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:09.697615 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
W20260928 22:45:09.697785 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:09.693692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35981 (local address 127.0.98.62:35459)
0928 22:45:09.693759 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:09.693760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:09.693770 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:09.693829 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:09.693829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:09.693839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:09.693891 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:09.694163 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.694654 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.695340 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.695529 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.695579 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.697718 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35981: BlockingRecv error: recv got EOF from 127.0.98.2:35981 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:09 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:09.821903 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:09.817631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:56973)
0928 22:45:09.817772 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:09.817782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:09.817835 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:09.817963 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:09.817963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:09.818113 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:09.818161 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.818162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.818762 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.818762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.819318 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:09.819320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.819619 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.819620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:09.819658 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.820114 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:09.820118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:09.821825 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
W20260928 22:45:09.822045 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:09.817718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56973 (local address 127.0.98.62:35459)
0928 22:45:09.817774 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:09.817775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:09.817831 (+ 56us) server_negotiation.cc:407] Connection header received
0928 22:45:09.817895 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:09.817896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:09.817907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:09.817946 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:09.818211 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.818705 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.819387 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:09.819565 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:09.819609 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:09.821974 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56973: BlockingRecv error: recv got EOF from 127.0.98.3:56973 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:10.586162 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:10.581415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:54353)
0928 22:45:10.581583 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:10.581595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:10.581653 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:10.581800 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:10.581800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:10.581962 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:10.582009 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.582010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.582634 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.582634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.583176 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.583178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.583528 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.583530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.583582 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.584076 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:10.584082 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:10.586093 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
W20260928 22:45:10.586263 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:10.581497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54353 (local address 127.0.98.62:35459)
0928 22:45:10.581610 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:10.581612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:10.581641 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:45:10.581708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:10.581708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:10.581718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:10.581787 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:10.582068 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.582572 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.583238 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.583427 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.583473 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.586191 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54353: BlockingRecv error: recv got EOF from 127.0.98.1:54353 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:10.702476 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:10.698104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:54423)
0928 22:45:10.698254 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:10.698267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:10.698344 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:10.698465 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:10.698466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:10.698623 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:10.698670 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.698671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.699320 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.699321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.699901 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.699903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.700230 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.700231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.700289 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.700696 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:10.700700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:10.702406 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
W20260928 22:45:10.702576 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:10.698197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54423 (local address 127.0.98.62:35459)
0928 22:45:10.698258 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:10.698259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:10.698338 (+ 79us) server_negotiation.cc:407] Connection header received
0928 22:45:10.698399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:10.698400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:10.698411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:10.698449 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:10.698724 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.699217 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.699970 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.700146 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.700191 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.702516 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54423: BlockingRecv error: recv got EOF from 127.0.98.2:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:10 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:10.826723 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:10.822313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:41327)
0928 22:45:10.822450 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:10.822460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:10.822519 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:10.822656 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:10.822657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:10.822821 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:10.822870 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.822871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.823598 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.823599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.824190 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:10.824193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.824499 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.824500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:10.824539 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.824996 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:10.825000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:10.826662 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260928 22:45:10.826831 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:10.822360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41327 (local address 127.0.98.62:35459)
0928 22:45:10.822403 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:10.822405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:10.822517 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:45:10.822560 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:10.822561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:10.822573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:10.822646 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:10.822939 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.823523 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.824250 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:10.824438 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:10.824490 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:10.826760 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41327: BlockingRecv error: recv got EOF from 127.0.98.3:41327 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:11.590911 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:11.586619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:52563)
0928 22:45:11.586763 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:11.586775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:11.586837 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:11.586979 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:11.586980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:11.587149 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:11.587194 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.587195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.587867 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.587869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.588387 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.588389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.588741 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.588742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.588804 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.589235 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:11.589240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:11.590852 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260928 22:45:11.591013 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:11.586675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52563 (local address 127.0.98.62:35459)
0928 22:45:11.586833 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:11.586834 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:11.586842 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:11.586894 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:11.586894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:11.586906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:11.586961 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:11.587264 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.587783 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.588462 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.588651 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.588703 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.590945 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52563: BlockingRecv error: recv got EOF from 127.0.98.1:52563 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:11.707628 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:11.703018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:48651)
0928 22:45:11.703176 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:11.703187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:11.703275 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:11.703427 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:11.703428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:11.703643 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:11.703690 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.703692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.704362 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.704363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.704931 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.704933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.705280 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.705281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.705335 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.705788 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:11.705792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:11.707548 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
W20260928 22:45:11.707695 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:11.703175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48651 (local address 127.0.98.62:35459)
0928 22:45:11.703242 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:11.703245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:11.703255 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:11.703344 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:11.703346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:11.703358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:11.703406 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:11.703778 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.704277 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.705018 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.705211 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.705268 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.707643 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48651: BlockingRecv error: recv got EOF from 127.0.98.2:48651 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:11 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:11.831158 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:11.827194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:46923)
0928 22:45:11.827350 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:11.827360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:11.827419 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:11.827533 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:11.827534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:11.827684 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:11.827737 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.827738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.828334 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.828335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.828877 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:11.828878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.829141 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.829141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:11.829176 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.829580 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:11.829584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:11.831099 (+ 1515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
W20260928 22:45:11.831247 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:11.827252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46923 (local address 127.0.98.62:35459)
0928 22:45:11.827312 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:11.827314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:11.827415 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:45:11.827470 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:11.827471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:11.827482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:11.827519 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:11.827793 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.828281 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.828925 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:11.829095 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:11.829137 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:11.831187 (+ 2050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46923: BlockingRecv error: recv got EOF from 127.0.98.3:46923 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:12.599855 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:12.591391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:42707)
0928 22:45:12.591517 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:12.591530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:12.591586 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:12.591757 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:12.591758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:12.591903 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:12.591950 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.591951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.592923 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.592924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.593452 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.593454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.593930 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.593931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.593980 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.597986 (+ 4006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:12.597991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:12.599797 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
W20260928 22:45:12.600070 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:12.591479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42707 (local address 127.0.98.62:35459)
0928 22:45:12.591614 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:12.591617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:12.591627 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:12.591653 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:12.591654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:12.591667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:12.591754 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:12.592009 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.592846 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.593528 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.593857 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.593925 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.599992 (+ 6067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42707: BlockingRecv error: recv got EOF from 127.0.98.1:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:12.712682 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:12.708131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:58967)
0928 22:45:12.708276 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:12.708288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:12.708346 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:12.708515 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:12.708516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:12.708681 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:12.708780 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.708782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.709519 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.709520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.710065 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.710066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.710462 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.710463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.710531 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.710963 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:12.710968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:12.712613 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260928 22:45:12.712815 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:12.708148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58967 (local address 127.0.98.62:35459)
0928 22:45:12.708191 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:12.708194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:12.708392 (+ 198us) server_negotiation.cc:407] Connection header received
0928 22:45:12.708423 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:12.708424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:12.708437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:12.708492 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:12.708832 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.709460 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.710167 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.710362 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.710427 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.712755 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58967: BlockingRecv error: recv got EOF from 127.0.98.2:58967 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:12 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:12.835808 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:12.831708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:42089)
0928 22:45:12.831859 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:12.831869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:12.831924 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:12.832053 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:12.832053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:12.832207 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:12.832253 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.832255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.832896 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.832897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.833430 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:12.833432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.833715 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.833715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:12.833753 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.834163 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:12.834167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:12.835748 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
W20260928 22:45:12.835908 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:12.831795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42089 (local address 127.0.98.62:35459)
0928 22:45:12.831855 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:12.831857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:12.831921 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:45:12.831985 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:12.831986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:12.831997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:12.832038 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:12.832309 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.832827 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.833490 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:12.833663 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:12.833705 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:12.835842 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42089: BlockingRecv error: recv got EOF from 127.0.98.3:42089 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:13.604820 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:13.600362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:53575)
0928 22:45:13.600560 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:13.600571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:13.600628 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:13.600766 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:13.600767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:13.600926 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:13.600976 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.600978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.601633 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.601634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.602160 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.602162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.602515 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.602516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.602551 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.603007 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:13.603012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:13.604754 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
W20260928 22:45:13.604923 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:13.600502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53575 (local address 127.0.98.62:35459)
0928 22:45:13.600629 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:13.600631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:13.600658 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:45:13.600679 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:13.600679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:13.600689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:13.600753 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:13.601052 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.601570 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.602233 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.602458 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.602509 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.604849 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53575: BlockingRecv error: recv got EOF from 127.0.98.1:53575 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:13.717797 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:13.713174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:40103)
0928 22:45:13.713299 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:13.713312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:13.713406 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:13.713540 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:13.713540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:13.713702 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:13.713749 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.713750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.714457 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.714458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.714983 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.714984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.715333 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.715334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.715388 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.715822 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:13.715826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:13.717740 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
W20260928 22:45:13.717913 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:13.713277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40103 (local address 127.0.98.62:35459)
0928 22:45:13.713332 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:13.713334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:13.713399 (+ 65us) server_negotiation.cc:407] Connection header received
0928 22:45:13.713462 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:13.713463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:13.713475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:13.713523 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:13.713846 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.714384 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.715070 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.715265 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.715322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.717840 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40103: BlockingRecv error: recv got EOF from 127.0.98.2:40103 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:13 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:13.840605 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:13.836323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:53741)
0928 22:45:13.836444 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:13.836458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:13.836526 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:13.836648 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:13.836649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:13.836823 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:13.836872 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.836874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.837490 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.837491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.838024 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:13.838026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.838339 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.838340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:13.838374 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.838787 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:13.838791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:13.840544 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
W20260928 22:45:13.840698 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:13.836425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53741 (local address 127.0.98.62:35459)
0928 22:45:13.836474 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:13.836476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:13.836509 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:45:13.836567 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:13.836568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:13.836579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:13.836624 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:13.836936 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.837429 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.838088 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:13.838282 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:13.838332 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:13.840638 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53741: BlockingRecv error: recv got EOF from 127.0.98.3:53741 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:14.609730 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:14.605317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:34521)
0928 22:45:14.605435 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:14.605446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:14.605495 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:14.605650 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:14.605650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:14.605803 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:14.605850 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.605852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.606520 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.606521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.607061 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.607063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.607372 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.607373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.607416 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.607866 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:14.607872 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:14.609668 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260928 22:45:14.609853 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:14.605411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34521 (local address 127.0.98.62:35459)
0928 22:45:14.605533 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:14.605535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:14.605543 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:14.605564 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:14.605565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:14.605575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:14.605636 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:14.605906 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.606427 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.607129 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.607313 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.607356 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.609784 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34521: BlockingRecv error: recv got EOF from 127.0.98.1:34521 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:14.722726 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:14.718360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:45561)
0928 22:45:14.718496 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:14.718510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:14.718581 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:14.718725 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:14.718726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:14.718938 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:14.718991 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.718993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.719669 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.719670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.720227 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.720229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.720570 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.720571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.720624 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.721058 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:14.721062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:14.722608 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260928 22:45:14.722831 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:14.718487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45561 (local address 127.0.98.62:35459)
0928 22:45:14.718548 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:14.718550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:14.718566 (+ 16us) server_negotiation.cc:407] Connection header received
0928 22:45:14.718642 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:14.718644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:14.718658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:14.718709 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:14.719063 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.719585 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.720311 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.720506 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.720562 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.722751 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45561: BlockingRecv error: recv got EOF from 127.0.98.2:45561 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:14 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:14.845413 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:14.841127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:34891)
0928 22:45:14.841310 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:14.841323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:14.841400 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:14.841518 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:14.841519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:14.841669 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:14.841718 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.841720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.842329 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.842330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.842861 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:14.842863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.843169 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.843170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:14.843212 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.843655 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:14.843659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:14.845334 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
W20260928 22:45:14.845489 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:14.841292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34891 (local address 127.0.98.62:35459)
0928 22:45:14.841352 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:14.841354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:14.841382 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:45:14.841452 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:14.841453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:14.841463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:14.841500 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:14.841771 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.842257 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.842925 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:14.843108 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:14.843154 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:14.845435 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34891: BlockingRecv error: recv got EOF from 127.0.98.3:34891 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:15.614598 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:15.610245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:55619)
0928 22:45:15.610378 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:15.610388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:15.610437 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:15.610650 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:15.610651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:15.610815 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:15.610863 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.610865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.611543 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.611543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.612074 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.612076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.612429 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.612430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.612467 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.612899 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:15.612904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:15.614536 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
W20260928 22:45:15.614702 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:15.610412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55619 (local address 127.0.98.62:35459)
0928 22:45:15.610526 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:15.610527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:15.610536 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:15.610559 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:15.610560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:15.610569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:15.610634 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:15.610961 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.611475 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.612150 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.612373 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.612426 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.614631 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55619: BlockingRecv error: recv got EOF from 127.0.98.1:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":49,"threads_started":1}
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:15.727792 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:15.723192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:60327)
0928 22:45:15.723339 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:15.723350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:15.723407 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:15.723534 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:15.723535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:15.723689 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:15.723737 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.723738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.724506 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.724507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.725123 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.725126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.725492 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.725493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.725536 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.725996 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:15.726000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:15.727707 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260928 22:45:15.727855 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:15.723256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60327 (local address 127.0.98.62:35459)
0928 22:45:15.723308 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:15.723310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:15.723404 (+ 94us) server_negotiation.cc:407] Connection header received
0928 22:45:15.723450 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:15.723451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:15.723463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:15.723522 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:15.723837 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.724431 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.725192 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.725393 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.725456 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.727797 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60327: BlockingRecv error: recv got EOF from 127.0.98.2:60327 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:15 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:15.850453 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:15.845945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:49469)
0928 22:45:15.846076 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:15.846086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:15.846149 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:15.846284 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:15.846285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:15.846453 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:15.846498 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.846500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.847138 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.847139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.847676 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:15.847678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.848007 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.848009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:15.848063 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.848550 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:15.848555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:15.850376 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20260928 22:45:15.850543 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:15.845993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49469 (local address 127.0.98.62:35459)
0928 22:45:15.846062 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:15.846065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:15.846124 (+ 59us) server_negotiation.cc:407] Connection header received
0928 22:45:15.846199 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:15.846200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:15.846213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:15.846261 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:15.846572 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.847058 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.847757 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:15.847943 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:15.847998 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:15.850487 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49469: BlockingRecv error: recv got EOF from 127.0.98.3:49469 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:16.619521 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:16.615055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:49213)
0928 22:45:16.615203 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:16.615217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:16.615290 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:16.615453 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:16.615455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:16.615615 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:16.615663 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.615665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.616413 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.616414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.617001 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.617004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.617353 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.617354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.617398 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.617821 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:16.617826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:16.619463 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
W20260928 22:45:16.619634 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:16.615126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49213 (local address 127.0.98.62:35459)
0928 22:45:16.615246 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:16.615248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:16.615272 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:45:16.615332 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:16.615332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:16.615342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:16.615398 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:16.615733 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.616321 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.617072 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.617292 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.617339 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.619558 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49213: BlockingRecv error: recv got EOF from 127.0.98.1:49213 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:16.732684 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:16.728271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:35663)
0928 22:45:16.728432 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:16.728442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:16.728496 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:16.728622 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:16.728622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:16.728805 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:16.728853 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.728854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.729447 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.729447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.729994 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.729996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.730319 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.730320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.730371 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.730773 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:16.730776 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:16.732615 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
W20260928 22:45:16.732848 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:16.728325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35663 (local address 127.0.98.62:35459)
0928 22:45:16.728377 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:16.728379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:16.728491 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:45:16.728539 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:16.728539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:16.728550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:16.728609 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:16.728904 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.729385 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.730061 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.730251 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.730303 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.732709 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35663: BlockingRecv error: recv got EOF from 127.0.98.2:35663 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:16 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:16.855304 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:16.850887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:38243)
0928 22:45:16.850999 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:16.851010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:16.851072 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:16.851190 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:16.851190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:16.851341 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:16.851394 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.851395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.852053 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.852054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.852584 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:16.852586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.852989 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.852990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:16.853029 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.853434 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:16.853439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:16.855233 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260928 22:45:16.855337 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:16.850950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38243 (local address 127.0.98.62:35459)
0928 22:45:16.851003 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:16.851005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:16.851064 (+ 59us) server_negotiation.cc:407] Connection header received
0928 22:45:16.851117 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:16.851117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:16.851129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:16.851177 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:16.851445 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.851978 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.852677 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:16.852929 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:16.852988 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:16.855290 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38243: BlockingRecv error: recv got EOF from 127.0.98.3:38243 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:17.630340 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:17.619982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:42431)
0928 22:45:17.620100 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:17.620111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:17.620159 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:17.620302 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:17.620302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:17.620455 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:17.620505 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.620507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.621164 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.621165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.621693 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.621695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.622044 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.622045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.622080 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.628302 (+ 6222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:17.628309 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:17.630271 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260928 22:45:17.630465 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:17.620059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42431 (local address 127.0.98.62:35459)
0928 22:45:17.620185 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:17.620187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:17.620195 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:17.620217 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:17.620217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:17.620227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:17.620291 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:17.620569 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.621108 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.621772 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.621981 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.622045 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.630385 (+ 8340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42431: BlockingRecv error: recv got EOF from 127.0.98.1:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:17.737576 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:17.733156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:60929)
0928 22:45:17.733304 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:17.733318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:17.733386 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:17.733517 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:17.733517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:17.733669 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:17.733719 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.733720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.734322 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.734323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.734842 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.734844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.735140 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.735141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.735177 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.735597 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:17.735601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:17.737504 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
W20260928 22:45:17.737660 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:17.733299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60929 (local address 127.0.98.62:35459)
0928 22:45:17.733359 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:17.733360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:17.733368 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:17.733447 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:17.733448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:17.733458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:17.733499 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:17.733774 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.734259 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.734915 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.735088 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.735131 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.737599 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60929: BlockingRecv error: recv got EOF from 127.0.98.2:60929 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:17 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:17.859929 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:17.855723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:54525)
0928 22:45:17.855830 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:17.855842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:17.855911 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:17.856038 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:17.856039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:17.856190 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:17.856240 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.856241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.856856 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.856857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.857399 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:17.857401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.857697 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.857697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:17.857742 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.858209 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:17.858213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:17.859869 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
W20260928 22:45:17.860026 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:17.855794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54525 (local address 127.0.98.62:35459)
0928 22:45:17.855854 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:17.855856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:17.855894 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:45:17.855963 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:17.855964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:17.855976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:17.856019 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:17.856289 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.856805 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.857461 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:17.857634 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:17.857679 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:17.859959 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54525: BlockingRecv error: recv got EOF from 127.0.98.3:54525 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:18.635360 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:18.630871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:41185)
0928 22:45:18.631013 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:18.631024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:18.631089 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:18.631221 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:18.631221 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:18.631379 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:18.631426 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.631428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.632128 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.632129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.632827 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.632831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.633156 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.633157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.633194 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.633615 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:18.633620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:18.635280 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
W20260928 22:45:18.635453 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:18.630958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41185 (local address 127.0.98.62:35459)
0928 22:45:18.631080 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:18.631081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:18.631091 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:45:18.631139 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:18.631139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:18.631149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:18.631206 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:18.631487 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.632058 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.632899 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.633098 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.633151 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.635392 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41185: BlockingRecv error: recv got EOF from 127.0.98.1:41185 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:18.742610 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:18.738081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:40711)
0928 22:45:18.738227 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:18.738238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:18.738294 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:18.738432 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:18.738432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:18.738584 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:18.738645 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.738647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.739307 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.739308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.739966 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.739968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.740299 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.740300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.740336 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.740769 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:18.740774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:18.742511 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260928 22:45:18.742681 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:18.738188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40711 (local address 127.0.98.62:35459)
0928 22:45:18.738267 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:18.738269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:18.738280 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:18.738352 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:18.738353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:18.738362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:18.738421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:18.738705 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.739219 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.740032 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.740234 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.740286 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.742628 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40711: BlockingRecv error: recv got EOF from 127.0.98.2:40711 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:18 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:18.865324 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:18.860398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.3:49471)
0928 22:45:18.860537 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:18.860547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:18.860597 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:18.860708 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:18.860709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:18.860949 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:18.861007 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.861010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.861699 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.861700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.862306 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:18.862309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.862661 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.862661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:18.862704 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.863159 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:18.863163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:18.865246 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
W20260928 22:45:18.865407 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:18.860420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49471 (local address 127.0.98.62:35459)
0928 22:45:18.860488 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:18.860490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:18.860591 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:45:18.860635 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:18.860636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:18.860647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:18.860695 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:18.861098 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.861594 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.862400 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:18.862585 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:18.862639 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:18.865352 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49471: BlockingRecv error: recv got EOF from 127.0.98.3:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 22:45:19 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:19 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:19.640409 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:19.635890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.1:41429)
0928 22:45:19.636040 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:19.636053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:19.636112 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:19.636236 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:19.636237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:19.636390 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:19.636438 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:19.636440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.637062 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.637063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:19.637598 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:19.637600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.638058 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.638059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:19.638111 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:19.638666 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:19.638672 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:19.640338 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
W20260928 22:45:19.640527 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:19.635962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41429 (local address 127.0.98.62:35459)
0928 22:45:19.636076 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:19.636077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:19.636098 (+ 21us) server_negotiation.cc:407] Connection header received
0928 22:45:19.636164 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:19.636164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:19.636174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:19.636218 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:19.636496 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.637010 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.637694 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.637901 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.637963 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:19.640443 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41429: BlockingRecv error: recv got EOF from 127.0.98.1:41429 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
Sep 28 22:45:19 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:45:19 dist-test-slave-pxr1 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:45:19.747382 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:19.743084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.98.2:42557)
0928 22:45:19.743200 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:19.743213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:19.743295 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:19.743426 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:19.743427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:19.743592 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:19.743637 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:19.743638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.744317 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.744318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:19.744872 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:19.744874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.745200 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.745201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:19.745281 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:19.745714 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:19.745718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:19.747325 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35459: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260928 22:45:19.747481 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:19.743172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42557 (local address 127.0.98.62:35459)
0928 22:45:19.743232 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:19.743234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:19.743276 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:45:19.743347 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:19.743348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:19.743360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:19.743409 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:19.743708 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.744238 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.744951 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.745137 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.745192 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:19.747422 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42557: BlockingRecv error: recv got EOF from 127.0.98.2:42557 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:45:19.795482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9100
I20260928 22:45:19.799535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9252
I20260928 22:45:19.803462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9384
I20260928 22:45:19.807163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9030
2026-09-28T22:45:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17273 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T22:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:19Z Disabled control of system clock
I20260928 22:45:19.832542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:45:19.912360 9581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:19.912585 9581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:19.912604 9581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:19.913893 9581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:19.913944 9581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:19.913956 9581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:19.913966 9581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:19.913991 9581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:19.915396 9581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:19.915643 9581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:19.915944 9581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:19.920893 9589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:19.920900 9586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:19.920974 9581 server_base.cc:1034] running on GCE node
W20260928 22:45:19.920940 9587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:19.921382 9581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:19.921717 9581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:19.922861 9581 hybrid_clock.cc:648] HybridClock initialized: now 1790635519922840 us; error 36 us; skew 500 ppm
I20260928 22:45:19.924062 9581 webserver.cc:460] Webserver started at http://127.0.98.62:38577/ using document root <none> and password file <none>
I20260928 22:45:19.924292 9581 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:19.924340 9581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:19.924463 9581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:19.925860 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "5369a603896c408da5e990c96ccfcf6d"
format_stamp: "Formatted at 2026-09-28 22:45:19 on dist-test-slave-pxr1"
server_key: "0561e2443177665eccbd9e91051ae4ce"
server_key_iv: "080bc8fe1cdffd365e234d0c1ecafc17"
server_key_version: "encryptionkey@0"
I20260928 22:45:19.926260 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "5369a603896c408da5e990c96ccfcf6d"
format_stamp: "Formatted at 2026-09-28 22:45:19 on dist-test-slave-pxr1"
server_key: "0561e2443177665eccbd9e91051ae4ce"
server_key_iv: "080bc8fe1cdffd365e234d0c1ecafc17"
server_key_version: "encryptionkey@0"
I20260928 22:45:19.928171 9581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:45:19.929142 9595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:19.929391 9581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:45:19.929504 9581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "5369a603896c408da5e990c96ccfcf6d"
format_stamp: "Formatted at 2026-09-28 22:45:19 on dist-test-slave-pxr1"
server_key: "0561e2443177665eccbd9e91051ae4ce"
server_key_iv: "080bc8fe1cdffd365e234d0c1ecafc17"
server_key_version: "encryptionkey@0"
I20260928 22:45:19.929584 9581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:19.947108 9581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:19.947430 9581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:19.947562 9581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:19.951123 9581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44617
I20260928 22:45:19.951179 9647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44617 every 8 connection(s)
I20260928 22:45:19.951493 9581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:19.952144 9648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:19.954723 9648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d: Bootstrap starting.
I20260928 22:45:19.955252 9648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:19.955466 9648 log.cc:824] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:19.956073 9648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d: No bootstrap required, opened a new log
I20260928 22:45:19.956861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9581
I20260928 22:45:19.956940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:45:19.957108 392 external_mini_cluster.cc:1428] Setting key 2f4bc86e1b5d4c74e697b4bb2f30cee4
I20260928 22:45:19.957967 9648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260928 22:45:19.958149 9648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:19.958182 9648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5369a603896c408da5e990c96ccfcf6d, State: Initialized, Role: FOLLOWER
I20260928 22:45:19.958318 9648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260928 22:45:19.958400 9648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:19.958429 9648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:19.958477 9648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:19.959019 9648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260928 22:45:19.959116 9648 leader_election.cc:302] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5369a603896c408da5e990c96ccfcf6d; no voters:
I20260928 22:45:19.959276 9648 leader_election.cc:288] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:45:19.959324 9653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:19.959431 9653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [term 1 LEADER]: Becoming Leader. State: Replica: 5369a603896c408da5e990c96ccfcf6d, State: Running, Role: LEADER
I20260928 22:45:19.959510 9653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260928 22:45:19.959543 9648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:19.959969 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5369a603896c408da5e990c96ccfcf6d. Latest consensus state: current_term: 1 leader_uuid: "5369a603896c408da5e990c96ccfcf6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260928 22:45:19.960047 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:19.960182 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5369a603896c408da5e990c96ccfcf6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5369a603896c408da5e990c96ccfcf6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260928 22:45:19.960230 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:19.960537 9659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:19.961047 9659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:45:19.962694 9659 catalog_manager.cc:1347] Generated new cluster ID: 61e6051536c44f65bacd6c4deba37963
I20260928 22:45:19.962749 9659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:19.963289 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:19.957701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.98.62:44617)
0928 22:45:19.957960 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:19.957969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:19.958007 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:45:19.958119 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:19.958126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:19.958198 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:19.958340 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:19.958569 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.960017 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.961231 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:19.961710 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:19.961817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:19.962477 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:19.962481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:19.962481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:19.962691 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:19.962735 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:19.962750 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:19.962807 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:19.963013 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 22:45:19.963090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260928 22:45:19.970592 9659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:45:19.971318 9659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:45:19.985162 9659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5369a603896c408da5e990c96ccfcf6d: Generated new TSK 0
I20260928 22:45:19.985373 9659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:45:19.990501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--builtin_ntp_servers=127.0.98.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 22:45:20.070785 9672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.071009 9672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.071031 9672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.072242 9672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.072290 9672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.072336 9672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:20.074292 9672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.074577 9672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.074895 9672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.080686 9677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.080823 9678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.080705 9680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.081210 9672 server_base.cc:1034] running on GCE node
I20260928 22:45:20.081413 9672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.081650 9672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.082813 9672 hybrid_clock.cc:648] HybridClock initialized: now 1790635520082797 us; error 30 us; skew 500 ppm
I20260928 22:45:20.084131 9672 webserver.cc:460] Webserver started at http://127.0.98.1:45911/ using document root <none> and password file <none>
I20260928 22:45:20.084362 9672 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.084410 9672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.084544 9672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.085641 9672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d48ea5bb30746c8a65498390078863b"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "9bca775d75ede8f42f093d77acfd15c0"
server_key_iv: "7beaab7a0a5d06abf3c428e822b7340c"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.085996 9672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d48ea5bb30746c8a65498390078863b"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "9bca775d75ede8f42f093d77acfd15c0"
server_key_iv: "7beaab7a0a5d06abf3c428e822b7340c"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.087268 9672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:45:20.087984 9686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.088166 9672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:20.088267 9672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "7d48ea5bb30746c8a65498390078863b"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "9bca775d75ede8f42f093d77acfd15c0"
server_key_iv: "7beaab7a0a5d06abf3c428e822b7340c"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.088339 9672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.105307 9672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.105583 9672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.105684 9672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.105878 9672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.106149 9672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.106177 9672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.106204 9672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.106220 9672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.111454 9672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32925
I20260928 22:45:20.111526 9799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32925 every 8 connection(s)
I20260928 22:45:20.111828 9672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:20.115499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9672
I20260928 22:45:20.115578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:45:20.115761 392 external_mini_cluster.cc:1428] Setting key b1e05d775fc7c2de0523175d86d73fea
I20260928 22:45:20.116686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--builtin_ntp_servers=127.0.98.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 22:45:20.117079 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.112557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42167 (local address 127.0.98.62:44617)
0928 22:45:20.112623 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.112625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.112920 (+ 295us) server_negotiation.cc:407] Connection header received
0928 22:45:20.113454 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.113455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.113471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.113522 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.114190 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.114693 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.115655 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.115867 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.115928 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.116832 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.116833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.116834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.116838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.116853 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.116857 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.116902 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.116965 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.117001 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:45:20.117350 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.112558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:42167)
0928 22:45:20.112797 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.112832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.113335 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.113596 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.113600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.113608 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.114087 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.114091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.114791 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.114792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.115558 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.115560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.116021 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.116022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.116136 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.116671 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.116671 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.116702 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.116754 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.116902 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.116906 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.116909 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.116970 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.117090 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260928 22:45:20.117769 9800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260928 22:45:20.117861 9800 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.118037 9800 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260928 22:45:20.118513 9612 ts_manager.cc:194] Registered new tserver with Master: 7d48ea5bb30746c8a65498390078863b (127.0.98.1:32925)
I20260928 22:45:20.119118 9612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42167
W20260928 22:45:20.198174 9803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.198446 9803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.198478 9803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.199674 9803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.199721 9803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.199756 9803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:20.201256 9803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.201496 9803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.201792 9803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.206638 9808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.206753 9803 server_base.cc:1034] running on GCE node
W20260928 22:45:20.206651 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.206643 9811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.207059 9803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.207289 9803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.208446 9803 hybrid_clock.cc:648] HybridClock initialized: now 1790635520208429 us; error 29 us; skew 500 ppm
I20260928 22:45:20.209746 9803 webserver.cc:460] Webserver started at http://127.0.98.2:34001/ using document root <none> and password file <none>
I20260928 22:45:20.210067 9803 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.210155 9803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.210294 9803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.211447 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "45718763a90740deb95ec5b59f63f917"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "c9e7dbcbfc2d8e87e1bf33b0dad91690"
server_key_iv: "7bd397bf73f20484db1346ef71dddcf1"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.211784 9803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45718763a90740deb95ec5b59f63f917"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "c9e7dbcbfc2d8e87e1bf33b0dad91690"
server_key_iv: "7bd397bf73f20484db1346ef71dddcf1"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.213027 9803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:45:20.213702 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.213842 9803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:20.213920 9803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "45718763a90740deb95ec5b59f63f917"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "c9e7dbcbfc2d8e87e1bf33b0dad91690"
server_key_iv: "7bd397bf73f20484db1346ef71dddcf1"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.213979 9803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.228404 9803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.228708 9803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.228927 9803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.229173 9803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.229491 9803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.229529 9803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.229559 9803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.229585 9803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.235164 9803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43603
I20260928 22:45:20.235217 9930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43603 every 8 connection(s)
I20260928 22:45:20.235522 9803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:45:20.240321 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.236187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51971 (local address 127.0.98.62:44617)
0928 22:45:20.236249 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.236251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.236489 (+ 238us) server_negotiation.cc:407] Connection header received
0928 22:45:20.237011 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.237012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.237020 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.237075 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.237682 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.238160 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.239093 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.239279 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.239324 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.240094 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.240095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.240096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.240100 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.240116 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.240120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.240171 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.240236 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.240262 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:45:20.240490 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.236187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:51971)
0928 22:45:20.236364 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.236405 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.236929 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.237139 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.237143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.237151 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.237621 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.237625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.238233 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.238234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.239028 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.239031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.239342 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.239342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.239449 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.239940 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.239940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.239969 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.239991 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.240176 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.240179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.240181 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.240229 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.240308 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 22:45:20.240984 9931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260928 22:45:20.241077 9931 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.241250 9931 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260928 22:45:20.241600 9612 ts_manager.cc:194] Registered new tserver with Master: 45718763a90740deb95ec5b59f63f917 (127.0.98.2:43603)
I20260928 22:45:20.241611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9803
I20260928 22:45:20.241683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:45:20.241852 392 external_mini_cluster.cc:1428] Setting key e3cdf1e1d607a4adcb95199af0f33cba
I20260928 22:45:20.242008 9612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51971
I20260928 22:45:20.242643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--builtin_ntp_servers=127.0.98.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 22:45:20.323263 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.323479 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.323500 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.324687 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.324764 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.324800 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:20.326205 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.326433 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.326745 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.331641 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.331633 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.331614 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.331668 9934 server_base.cc:1034] running on GCE node
I20260928 22:45:20.331918 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.332132 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.333293 9934 hybrid_clock.cc:648] HybridClock initialized: now 1790635520333279 us; error 30 us; skew 500 ppm
I20260928 22:45:20.334656 9934 webserver.cc:460] Webserver started at http://127.0.98.3:36939/ using document root <none> and password file <none>
I20260928 22:45:20.334880 9934 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.334925 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.335057 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.336158 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "7dafae425d564d7f94b76bddf15a325a"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "becd07c1ee7aed30552bc683bf99698b"
server_key_iv: "de150d05681521bb639a200c12945657"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.336491 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7dafae425d564d7f94b76bddf15a325a"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "becd07c1ee7aed30552bc683bf99698b"
server_key_iv: "de150d05681521bb639a200c12945657"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.337812 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:45:20.338513 9948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.338685 9934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:20.338761 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "7dafae425d564d7f94b76bddf15a325a"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "becd07c1ee7aed30552bc683bf99698b"
server_key_iv: "de150d05681521bb639a200c12945657"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.338831 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.372398 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.372705 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.372874 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.373106 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.373428 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.373461 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.373489 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.373507 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.378772 9934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33571
I20260928 22:45:20.378822 10061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33571 every 8 connection(s)
I20260928 22:45:20.379135 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:45:20.384132 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.379800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35143 (local address 127.0.98.62:44617)
0928 22:45:20.379843 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.379845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.380072 (+ 227us) server_negotiation.cc:407] Connection header received
0928 22:45:20.380569 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.380570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.380583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.380634 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.381309 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.381811 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.382769 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.382992 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.383050 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.383847 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.383849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.383849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.383854 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.383870 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.383873 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.383919 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.384019 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.384051 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:45:20.384313 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.379793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:35143)
0928 22:45:20.379948 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.379987 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.380472 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.380701 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.380704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.380713 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.381221 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.381225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.381901 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.381901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.382680 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:20.382682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:20.383075 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:20.383076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:20.383199 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:20.383712 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.383712 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.383742 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.383765 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.383928 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.383945 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.383948 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.384015 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.384120 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260928 22:45:20.384750 10062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260928 22:45:20.384850 10062 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.385023 10062 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260928 22:45:20.385329 9612 ts_manager.cc:194] Registered new tserver with Master: 7dafae425d564d7f94b76bddf15a325a (127.0.98.3:33571)
I20260928 22:45:20.385684 9612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35143
I20260928 22:45:20.387434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 9934
I20260928 22:45:20.387522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:45:20.387702 392 external_mini_cluster.cc:1428] Setting key 94e72debc450c71a7f01eca995b343a1
I20260928 22:45:20.388470 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:45:20.390224 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:20.389751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.0.98.62:44617)
0928 22:45:20.389811 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.389813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.389911 (+ 98us) server_negotiation.cc:407] Connection header received
0928 22:45:20.389975 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.389976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.389989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.390036 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.390157 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47028: BlockingRecv error: recv got EOF from 127.0.0.1:47028 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:45:20.390268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9672
I20260928 22:45:20.394676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9803
I20260928 22:45:20.398335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9934
I20260928 22:45:20.401984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 9581
2026-09-28T22:45:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (593 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T22:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:20Z Disabled control of system clock
I20260928 22:45:20.426882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37549
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:45:20.503881 10073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.504129 10073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.504149 10073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.505290 10073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:20.505326 10073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.505336 10073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.505347 10073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:20.505358 10073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:20.506637 10073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.506830 10073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.507071 10073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.511775 10081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.511833 10073 server_base.cc:1034] running on GCE node
W20260928 22:45:20.511790 10078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.511780 10079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.512184 10073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.512475 10073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.513618 10073 hybrid_clock.cc:648] HybridClock initialized: now 1790635520513604 us; error 30 us; skew 500 ppm
I20260928 22:45:20.514830 10073 webserver.cc:460] Webserver started at http://127.0.98.62:44351/ using document root <none> and password file <none>
I20260928 22:45:20.515053 10073 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.515103 10073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.515230 10073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.516291 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "f709ed59af214642a769c3e09b964810"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "19bf8042b0efbde9fad71900edab447b"
server_key_iv: "4268fc267ec064a8466f7edc82611389"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.516623 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "f709ed59af214642a769c3e09b964810"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "19bf8042b0efbde9fad71900edab447b"
server_key_iv: "4268fc267ec064a8466f7edc82611389"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.517895 10073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:20.518556 10087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.518718 10073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:20.518803 10073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "f709ed59af214642a769c3e09b964810"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "19bf8042b0efbde9fad71900edab447b"
server_key_iv: "4268fc267ec064a8466f7edc82611389"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.518868 10073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.522280 10073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.522547 10073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.522661 10073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.526265 10073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37549
I20260928 22:45:20.526297 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37549 every 8 connection(s)
I20260928 22:45:20.526602 10073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:20.527197 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:20.530041 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810: Bootstrap starting.
I20260928 22:45:20.530651 10140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:20.530704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10073
I20260928 22:45:20.530797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:45:20.530989 392 external_mini_cluster.cc:1428] Setting key 3395aa689ac597c3d0fd332ac7816e51
I20260928 22:45:20.530989 10140 log.cc:824] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:20.531891 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810: No bootstrap required, opened a new log
I20260928 22:45:20.533406 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.531479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.0.98.62:37549)
0928 22:45:20.531643 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.531650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.531676 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:45:20.532142 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.532150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.532190 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.532298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.532983 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.532987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.532988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.532992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.533034 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.533044 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.533109 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.533174 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.533221 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 22:45:20.533629 10140 raft_consensus.cc:348] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } }
I20260928 22:45:20.533798 10140 raft_consensus.cc:374] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:20.533823 10140 raft_consensus.cc:729] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f709ed59af214642a769c3e09b964810, State: Initialized, Role: FOLLOWER
I20260928 22:45:20.533941 10140 consensus_queue.cc:260] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } }
I20260928 22:45:20.534022 10140 raft_consensus.cc:388] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:20.534046 10140 raft_consensus.cc:482] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:20.534092 10140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:20.534937 10140 raft_consensus.cc:504] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } }
I20260928 22:45:20.535027 10140 leader_election.cc:302] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f709ed59af214642a769c3e09b964810; no voters:
I20260928 22:45:20.535181 10140 leader_election.cc:288] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:45:20.535248 10145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:20.535391 10145 raft_consensus.cc:686] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [term 1 LEADER]: Becoming Leader. State: Replica: f709ed59af214642a769c3e09b964810, State: Running, Role: LEADER
I20260928 22:45:20.535427 10140 sys_catalog.cc:564] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:20.535496 10145 consensus_queue.cc:237] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } }
I20260928 22:45:20.535867 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f709ed59af214642a769c3e09b964810. Latest consensus state: current_term: 1 leader_uuid: "f709ed59af214642a769c3e09b964810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } } }
I20260928 22:45:20.535957 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:20.536178 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:20.536206 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f709ed59af214642a769c3e09b964810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f709ed59af214642a769c3e09b964810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37549 } } }
I20260928 22:45:20.536340 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:20.536660 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:45:20.538066 10153 catalog_manager.cc:1347] Generated new cluster ID: d6ceb2022f794bac85223395a345b6b0
I20260928 22:45:20.538118 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:20.555357 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:45:20.555532 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:45:20.562397 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f709ed59af214642a769c3e09b964810: Generated new TSK 0
I20260928 22:45:20.562575 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:45:20.566908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:45:20.646334 10164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.646538 10164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.646559 10164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.647750 10164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.647794 10164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.647818 10164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:20.649246 10164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.649478 10164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.649771 10164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.655457 10170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.655467 10172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.655579 10169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.655826 10164 server_base.cc:1034] running on GCE node
I20260928 22:45:20.656015 10164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.656268 10164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.657415 10164 hybrid_clock.cc:648] HybridClock initialized: now 1790635520657404 us; error 26 us; skew 500 ppm
I20260928 22:45:20.658617 10164 webserver.cc:460] Webserver started at http://127.0.98.1:43901/ using document root <none> and password file <none>
I20260928 22:45:20.658854 10164 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.658898 10164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.659017 10164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.660081 10164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "a6c65ee1f3a64fde9c0e68581daec81f"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "36141c5e6f3206b71944d31019de2b30"
server_key_iv: "cecdefe4050c488def1190e81ec31131"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.660408 10164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a6c65ee1f3a64fde9c0e68581daec81f"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "36141c5e6f3206b71944d31019de2b30"
server_key_iv: "cecdefe4050c488def1190e81ec31131"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.661715 10164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:45:20.662415 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.662556 10164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:20.662632 10164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "a6c65ee1f3a64fde9c0e68581daec81f"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "36141c5e6f3206b71944d31019de2b30"
server_key_iv: "cecdefe4050c488def1190e81ec31131"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.662700 10164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.666020 10164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.666276 10164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.666390 10164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.666587 10164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.666863 10164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.666894 10164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.666921 10164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.666944 10164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.672363 10164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37051
I20260928 22:45:20.672403 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37051 every 8 connection(s)
I20260928 22:45:20.672693 10164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:20.675531 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.673355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51633 (local address 127.0.98.62:37549)
0928 22:45:20.673395 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.673397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.673652 (+ 255us) server_negotiation.cc:407] Connection header received
0928 22:45:20.674125 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.674126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.674133 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.674184 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.675294 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.675296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.675296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.675301 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.675316 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.675321 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.675369 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.675448 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.675462 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:45:20.675719 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.673346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37549 (local address 127.0.98.1:51633)
0928 22:45:20.673518 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.673551 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.674019 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.674253 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.674257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.674267 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.675145 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.675146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.675173 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.675201 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.675376 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.675381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.675384 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.675451 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.675515 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 22:45:20.676184 10292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37549
I20260928 22:45:20.676270 10292 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.676381 10292 heartbeater.cc:511] Master 127.0.98.62:37549 requested a full tablet report, sending...
I20260928 22:45:20.676867 10104 ts_manager.cc:194] Registered new tserver with Master: a6c65ee1f3a64fde9c0e68581daec81f (127.0.98.1:37051)
I20260928 22:45:20.681764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10164
I20260928 22:45:20.681839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:45:20.681981 392 external_mini_cluster.cc:1428] Setting key 1c3e367445182c9d336ef93a33f4011a
I20260928 22:45:20.682776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:45:20.762176 10295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.762394 10295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.762423 10295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.763633 10295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.763685 10295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.763720 10295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:20.765218 10295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.765465 10295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.765753 10295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.770550 10301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.770565 10303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.770555 10300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.770684 10295 server_base.cc:1034] running on GCE node
I20260928 22:45:20.770967 10295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.771206 10295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.772354 10295 hybrid_clock.cc:648] HybridClock initialized: now 1790635520772338 us; error 28 us; skew 500 ppm
I20260928 22:45:20.773627 10295 webserver.cc:460] Webserver started at http://127.0.98.2:42379/ using document root <none> and password file <none>
I20260928 22:45:20.773838 10295 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.773886 10295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.774014 10295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.775039 10295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "f978f983de614d2ebdcc5aaeb248f129"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "86d09dd529a1bf74ebcca27534d5c152"
server_key_iv: "bec417737b79b22723d096ad663dacdb"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.775375 10295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f978f983de614d2ebdcc5aaeb248f129"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "86d09dd529a1bf74ebcca27534d5c152"
server_key_iv: "bec417737b79b22723d096ad663dacdb"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.776583 10295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:45:20.777328 10309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.777472 10295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.777551 10295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "f978f983de614d2ebdcc5aaeb248f129"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "86d09dd529a1bf74ebcca27534d5c152"
server_key_iv: "bec417737b79b22723d096ad663dacdb"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.777617 10295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.780817 10295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.781069 10295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.781198 10295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.781397 10295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.781675 10295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.781708 10295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.781734 10295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.781760 10295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.787129 10295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41365
I20260928 22:45:20.787176 10422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41365 every 8 connection(s)
I20260928 22:45:20.787485 10295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:45:20.790282 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.788180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38459 (local address 127.0.98.62:37549)
0928 22:45:20.788231 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.788233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.788453 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:45:20.788945 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.788946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.788951 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.789002 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.790071 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.790072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.790072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.790076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.790099 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.790102 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.790147 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.790214 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.790226 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:45:20.790455 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.788158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37549 (local address 127.0.98.2:38459)
0928 22:45:20.788334 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.788367 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.788864 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.789059 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.789062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.789071 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.789929 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.789929 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.789958 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.789987 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.790164 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.790166 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.790168 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.790220 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.790269 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 22:45:20.790875 10423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37549
I20260928 22:45:20.790956 10423 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.791100 10423 heartbeater.cc:511] Master 127.0.98.62:37549 requested a full tablet report, sending...
I20260928 22:45:20.791430 10104 ts_manager.cc:194] Registered new tserver with Master: f978f983de614d2ebdcc5aaeb248f129 (127.0.98.2:41365)
I20260928 22:45:20.797004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10295
I20260928 22:45:20.797096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:45:20.797245 392 external_mini_cluster.cc:1428] Setting key acfab7ff038b955ec1e6885f1effeb78
I20260928 22:45:20.798110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:45:20.876873 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:20.877085 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:20.877107 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:20.878228 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:20.878262 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:20.878286 10426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:20.879575 10426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:20.879781 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:20.880060 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:20.884883 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.884902 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:20.884902 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:20.885040 10426 server_base.cc:1034] running on GCE node
I20260928 22:45:20.885290 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:20.885527 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:20.886667 10426 hybrid_clock.cc:648] HybridClock initialized: now 1790635520886655 us; error 29 us; skew 500 ppm
I20260928 22:45:20.887874 10426 webserver.cc:460] Webserver started at http://127.0.98.3:35529/ using document root <none> and password file <none>
I20260928 22:45:20.888083 10426 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:20.888128 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:20.888247 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:20.889323 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "172126c29a7a4858b9225b2b81b3b5b9"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "d5ba0fbf8f684d138e15d4ac4e38035d"
server_key_iv: "93f6cb90fd52edd22a6a996e19f72b9e"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.889652 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "172126c29a7a4858b9225b2b81b3b5b9"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "d5ba0fbf8f684d138e15d4ac4e38035d"
server_key_iv: "93f6cb90fd52edd22a6a996e19f72b9e"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.890805 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:45:20.891441 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.891613 10426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:20.891709 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "172126c29a7a4858b9225b2b81b3b5b9"
format_stamp: "Formatted at 2026-09-28 22:45:20 on dist-test-slave-pxr1"
server_key: "d5ba0fbf8f684d138e15d4ac4e38035d"
server_key_iv: "93f6cb90fd52edd22a6a996e19f72b9e"
server_key_version: "encryptionkey@0"
I20260928 22:45:20.891768 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:20.895108 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:20.895375 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:20.895498 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:20.895695 10426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:20.895996 10426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:20.896028 10426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.896054 10426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:20.896071 10426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:20.901590 10426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45529
I20260928 22:45:20.901631 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45529 every 8 connection(s)
I20260928 22:45:20.901958 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:45:20.902379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10426
I20260928 22:45:20.902474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:45:20.902652 392 external_mini_cluster.cc:1428] Setting key ff902595a5426739a43ffe8664122977
I20260928 22:45:20.905479 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.903229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41801 (local address 127.0.98.62:37549)
0928 22:45:20.903477 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.903479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.903561 (+ 82us) server_negotiation.cc:407] Connection header received
0928 22:45:20.904045 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.904046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.904052 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.904108 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.905243 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:20.905245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:20.905245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:45:20.905249 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:20.905264 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:45:20.905268 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:20.905315 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:20.905398 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:45:20.905411 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260928 22:45:20.905658 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:20.903251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37549 (local address 127.0.98.3:41801)
0928 22:45:20.903440 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:20.903480 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:20.903955 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:20.904182 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:20.904185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:20.904194 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:20.905103 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:45:20.905104 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:20.905132 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:45:20.905161 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:20.905331 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:20.905336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:20.905339 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:45:20.905402 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:45:20.905465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 22:45:20.906018 10554 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37549
I20260928 22:45:20.906098 10554 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:20.906248 10554 heartbeater.cc:511] Master 127.0.98.62:37549 requested a full tablet report, sending...
I20260928 22:45:20.906546 10104 ts_manager.cc:194] Registered new tserver with Master: 172126c29a7a4858b9225b2b81b3b5b9 (127.0.98.3:45529)
I20260928 22:45:20.914376 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:45:20.916054 10144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:45:20.915591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.0.98.62:37549)
0928 22:45:20.915647 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:20.915649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:20.915750 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:45:20.915810 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:20.915811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:20.915816 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:20.915868 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:20.915987 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57190: BlockingRecv error: recv got EOF from 127.0.0.1:57190 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:45:20.916093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10164
I20260928 22:45:20.920053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10295
I20260928 22:45:20.923928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10426
I20260928 22:45:20.927314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10073
2026-09-28T22:45:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:45:20 dist-test-slave-pxr1 krb5kdc[10564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:45:20 dist-test-slave-pxr1 krb5kdc[10564](info): set up 2 sockets
Sep 28 22:45:20 dist-test-slave-pxr1 krb5kdc[10564](info): commencing operation
krb5kdc: starting...
W20260928 22:45:21.877363 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:45:21 dist-test-slave-pxr1 krb5kdc[10564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:45:21.992376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:22.071453 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:22.071666 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:22.071688 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:22.072837 10577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:45:22.072876 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:22.072888 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:22.072899 10577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:45:22.072911 10577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:45:22.074380 10577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:22.074587 10577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:22.074762 10577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:22.079393 10585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.079483 10577 server_base.cc:1034] running on GCE node
W20260928 22:45:22.079355 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:22.079576 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.079799 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:22.080066 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:22.081203 10577 hybrid_clock.cc:648] HybridClock initialized: now 1790635522081189 us; error 31 us; skew 500 ppm
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:22.083495 10577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:45:22.083971 10577 webserver.cc:460] Webserver started at http://127.0.98.62:41501/ using document root <none> and password file <none>
I20260928 22:45:22.084184 10577 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:22.084232 10577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:22.084352 10577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:22.085450 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "6f43369d2a434b1a84c1b16599227e38"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "d2c7d4132d385e5932d35008c4bf0ef2"
server_key_iv: "b3e46bbaa3d66082d2d58ce10f450200"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.085786 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f43369d2a434b1a84c1b16599227e38"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "d2c7d4132d385e5932d35008c4bf0ef2"
server_key_iv: "b3e46bbaa3d66082d2d58ce10f450200"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.087076 10577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:45:22.087728 10592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.087880 10577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:22.087962 10577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "6f43369d2a434b1a84c1b16599227e38"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "d2c7d4132d385e5932d35008c4bf0ef2"
server_key_iv: "b3e46bbaa3d66082d2d58ce10f450200"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.088021 10577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:22.099051 10577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:22.100924 10577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:22.101070 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:22.104811 10577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45991
I20260928 22:45:22.104854 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45991 every 8 connection(s)
I20260928 22:45:22.105208 10577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:45:22.106150 10645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:22.106998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10577
I20260928 22:45:22.107086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:45:22.107264 392 external_mini_cluster.cc:1428] Setting key f8edfe390712747318f97a22ee9524d8
I20260928 22:45:22.108958 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: Bootstrap starting.
I20260928 22:45:22.109591 10645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:22.109884 10645 log.cc:824] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:22.110639 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: No bootstrap required, opened a new log
I20260928 22:45:22.112846 10645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260928 22:45:22.113008 10645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:22.113032 10645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f43369d2a434b1a84c1b16599227e38, State: Initialized, Role: FOLLOWER
I20260928 22:45:22.113148 10645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260928 22:45:22.113236 10645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:45:22.113278 10645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:45:22.113323 10645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:22.113940 10645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260928 22:45:22.114051 10645 leader_election.cc:302] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f43369d2a434b1a84c1b16599227e38; no voters:
I20260928 22:45:22.114220 10645 leader_election.cc:288] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:45:22.114260 10650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:22.114419 10650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [term 1 LEADER]: Becoming Leader. State: Replica: 6f43369d2a434b1a84c1b16599227e38, State: Running, Role: LEADER
I20260928 22:45:22.114490 10645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:45:22.114514 10650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260928 22:45:22.114897 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f43369d2a434b1a84c1b16599227e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260928 22:45:22.114897 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f43369d2a434b1a84c1b16599227e38. Latest consensus state: current_term: 1 leader_uuid: "6f43369d2a434b1a84c1b16599227e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43369d2a434b1a84c1b16599227e38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260928 22:45:22.115000 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:22.115002 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38 [sys.catalog]: This master's current role is: LEADER
I20260928 22:45:22.115741 10658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:45:22.116629 10658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:45:22.118201 10658 catalog_manager.cc:1347] Generated new cluster ID: 4a2d617777cc4dd697c4c71cd5e24cd4
I20260928 22:45:22.118247 10658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:45:22.119043 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.107794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.0.98.62:45991)
0928 22:45:22.108003 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.108013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.108048 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:45:22.108665 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.108675 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.108766 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.108919 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:22.109512 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.110795 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.111546 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.112173 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.112297 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.115191 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:22.115196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:22.115196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:22.115210 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:22.117247 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.117663 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.117667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.117667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.117785 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.118020 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.118021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.118022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.118322 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:22.118472 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.118585 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.118830 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 22:45:22.137379 10658 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:45:22.137956 10658 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:45:22.148159 10658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: Generated new TSK 0
I20260928 22:45:22.148353 10658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:45:22.176529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:22.256279 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:22.256479 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:22.256503 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:22.257705 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:22.257751 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:22.257777 10671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:45:22.259239 10671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:22.259487 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:22.259733 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:22.264554 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:22.264569 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.264744 10671 server_base.cc:1034] running on GCE node
W20260928 22:45:22.264853 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.265079 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:22.265317 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:22.266460 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790635522266451 us; error 31 us; skew 500 ppm
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:22.268950 10671 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:45:22.269359 10671 webserver.cc:460] Webserver started at http://127.0.98.1:39907/ using document root <none> and password file <none>
I20260928 22:45:22.269577 10671 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:22.269624 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:22.269747 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:22.270769 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ea702edf30e4c1ba1b6d55d808f0243"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "120d2162cb4ab4bc37206a072328bd33"
server_key_iv: "c25701383080555ab850d02d9251a094"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.271093 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ea702edf30e4c1ba1b6d55d808f0243"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "120d2162cb4ab4bc37206a072328bd33"
server_key_iv: "c25701383080555ab850d02d9251a094"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.272301 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:45:22.272967 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.273116 10671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:45:22.273197 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "1ea702edf30e4c1ba1b6d55d808f0243"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "120d2162cb4ab4bc37206a072328bd33"
server_key_iv: "c25701383080555ab850d02d9251a094"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.273257 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:22.310962 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:22.313087 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:22.313223 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:22.313437 10671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:22.313738 10671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:22.313771 10671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.313799 10671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:22.313824 10671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.320183 10671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43713
I20260928 22:45:22.320223 10799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43713 every 8 connection(s)
I20260928 22:45:22.320533 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:45:22.321925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10671
I20260928 22:45:22.322022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:45:22.322218 392 external_mini_cluster.cc:1428] Setting key 38270b48e1609e961d0a402d09029719
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:45:22.332494 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.321298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44745 (local address 127.0.98.62:45991)
0928 22:45:22.321380 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.321382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.321532 (+ 150us) server_negotiation.cc:407] Connection header received
0928 22:45:22.322180 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.322181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.322194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.322222 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:22.324021 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.324816 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.326317 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.326631 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.326693 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.329509 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:22.329511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:22.329511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:22.329519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:22.331305 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.331717 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.331718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.331718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.331777 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.331969 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.331970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.331970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.332073 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:22.332116 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.332348 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.332425 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 22:45:22.332782 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.321268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45991 (local address 127.0.98.1:44745)
0928 22:45:22.321414 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.321450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.322051 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.322846 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.322851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.323602 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:22.323926 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.323932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.324923 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.324924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.326214 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.326217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.326769 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.326771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.326922 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.327537 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:22.327548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:22.329360 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:22.331411 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.331419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.331421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.331641 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.331839 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.331839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.331839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.331930 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.332107 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:22.332110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:22.332232 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.332309 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.332440 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 22:45:22.333318 10800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45991
I20260928 22:45:22.333420 10800 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:22.333606 10800 heartbeater.cc:511] Master 127.0.98.62:45991 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 22:45:22.334168 10609 ts_manager.cc:194] Registered new tserver with Master: 1ea702edf30e4c1ba1b6d55d808f0243 (127.0.98.1:43713)
I20260928 22:45:22.334787 10609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:45:22.347833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:22.426736 10805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:22.426954 10805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:22.426977 10805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:22.428162 10805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:22.428211 10805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:22.428237 10805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:45:22.429740 10805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:22.429972 10805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:22.430197 10805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:22.435040 10813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:22.435045 10810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.435179 10805 server_base.cc:1034] running on GCE node
W20260928 22:45:22.435045 10811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.435482 10805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:22.435706 10805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:22.436839 10805 hybrid_clock.cc:648] HybridClock initialized: now 1790635522436814 us; error 40 us; skew 500 ppm
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:22.439419 10805 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:45:22.439831 10805 webserver.cc:460] Webserver started at http://127.0.98.2:34211/ using document root <none> and password file <none>
I20260928 22:45:22.440006 10805 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:22.440044 10805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:22.440137 10805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:22.441190 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8cfee0f4aa6487684b2fa8eba9cf544"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "45fe27960cc8a821f417e4904f2c2f07"
server_key_iv: "3b2d3c6be79d47217319452bf0b193e6"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.441495 10805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8cfee0f4aa6487684b2fa8eba9cf544"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "45fe27960cc8a821f417e4904f2c2f07"
server_key_iv: "3b2d3c6be79d47217319452bf0b193e6"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.442734 10805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:45:22.443365 10820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.443524 10805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:45:22.443606 10805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "f8cfee0f4aa6487684b2fa8eba9cf544"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "45fe27960cc8a821f417e4904f2c2f07"
server_key_iv: "3b2d3c6be79d47217319452bf0b193e6"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.443667 10805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:22.455257 10805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:22.457116 10805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:22.457257 10805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:22.457480 10805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:22.457772 10805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:22.457805 10805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.457834 10805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:22.457850 10805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.463325 10805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38899
I20260928 22:45:22.463397 10933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38899 every 8 connection(s)
I20260928 22:45:22.463685 10805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:22.472404 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.464401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48631 (local address 127.0.98.62:45991)
0928 22:45:22.464486 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.464488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.464641 (+ 153us) server_negotiation.cc:407] Connection header received
0928 22:45:22.465179 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.465180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.465190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.465239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:22.466193 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.466694 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.467674 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.467875 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.467932 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.470041 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:22.470042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:22.470043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:22.470050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:22.471166 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.471557 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.471559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.471559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.471620 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.471825 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.471825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.471826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.471934 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:22.471983 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.472215 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.472301 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:45:22.472438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10805
I20260928 22:45:22.472486 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.464375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45991 (local address 127.0.98.2:48631)
0928 22:45:22.464520 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.464559 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.465091 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.465297 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.465301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.465867 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:22.466077 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.466083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.466772 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.466772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.467587 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.467590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.467945 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.467945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.468051 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.468554 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:22.468562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:22.469954 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:22.471237 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.471243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.471245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.471462 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.471682 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.471683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.471683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.471775 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.471979 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:22.471982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:22.472112 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.472193 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.472312 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260928 22:45:22.472527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:45:22.472703 392 external_mini_cluster.cc:1428] Setting key 6fd40dbc26e2820bde3dceba6506052d
I20260928 22:45:22.473213 10934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45991
I20260928 22:45:22.473316 10934 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:22.473510 10934 heartbeater.cc:511] Master 127.0.98.62:45991 requested a full tablet report, sending...
I20260928 22:45:22.473912 10608 ts_manager.cc:194] Registered new tserver with Master: f8cfee0f4aa6487684b2fa8eba9cf544 (127.0.98.2:38899)
I20260928 22:45:22.474295 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48631
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:45:22.496153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:45:22.575112 10939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:45:22.575320 10939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:45:22.575353 10939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:45:22.576552 10939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:45:22.576598 10939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:45:22.576633 10939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:45:22.578161 10939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:45:22.578405 10939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:45:22.578611 10939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:45:22.583636 10945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:45:22.583636 10944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.583796 10939 server_base.cc:1034] running on GCE node
W20260928 22:45:22.583630 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:45:22.584062 10939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:45:22.584295 10939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:45:22.585434 10939 hybrid_clock.cc:648] HybridClock initialized: now 1790635522585413 us; error 38 us; skew 500 ppm
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:45:22.588049 10939 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:45:22.588459 10939 webserver.cc:460] Webserver started at http://127.0.98.3:45489/ using document root <none> and password file <none>
I20260928 22:45:22.588667 10939 fs_manager.cc:359] Metadata directory not provided
I20260928 22:45:22.588708 10939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:45:22.588855 10939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:45:22.589929 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "66b0d00006d84abe8e0f83952ef372c1"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "c02d15dba552017991fea6e7c6aa283d"
server_key_iv: "fd025967672446ec88a2418e48fa2b8e"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.590260 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66b0d00006d84abe8e0f83952ef372c1"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "c02d15dba552017991fea6e7c6aa283d"
server_key_iv: "fd025967672446ec88a2418e48fa2b8e"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.591724 10939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:45:22.592518 10954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.592687 10939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:45:22.592792 10939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "66b0d00006d84abe8e0f83952ef372c1"
format_stamp: "Formatted at 2026-09-28 22:45:22 on dist-test-slave-pxr1"
server_key: "c02d15dba552017991fea6e7c6aa283d"
server_key_iv: "fd025967672446ec88a2418e48fa2b8e"
server_key_version: "encryptionkey@0"
I20260928 22:45:22.592859 10939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:45:22.606279 10939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:45:22.608250 10939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:45:22.608364 10939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:45:22.608549 10939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:45:22.608871 10939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:45:22.608915 10939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.608938 10939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:45:22.608949 10939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:45:22.614327 10939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32769
I20260928 22:45:22.614400 11067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32769 every 8 connection(s)
I20260928 22:45:22.614698 10939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:45:22.620443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 10939
I20260928 22:45:22.620525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
Sep 28 22:45:22 dist-test-slave-pxr1 krb5kdc[10564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:45:22.620687 392 external_mini_cluster.cc:1428] Setting key ea073ff18f782b53bbd48ccdec800217
I20260928 22:45:22.623973 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.615438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41061 (local address 127.0.98.62:45991)
0928 22:45:22.615483 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.615485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.615714 (+ 229us) server_negotiation.cc:407] Connection header received
0928 22:45:22.616269 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.616270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.616282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.616332 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:22.617293 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.617807 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.618807 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.619010 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.619063 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.621229 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:22.621230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:22.621231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:22.621238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:22.622481 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.623011 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.623012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.623013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.623069 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.623326 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.623327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.623327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.623446 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:22.623500 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.623767 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.623862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:45:22.624125 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.615409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45991 (local address 127.0.98.3:41061)
0928 22:45:22.615570 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.615612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.616170 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.616404 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.616408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.616993 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:45:22.617192 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.617197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.617910 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.617911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.618703 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.618705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.619087 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.619087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.619197 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.619648 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:45:22.619656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:45:22.621152 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:45:22.622560 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.622567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.622569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.622914 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.623130 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:45:22.623131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:45:22.623132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:45:22.623267 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:45:22.623503 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:45:22.623509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:45:22.623645 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.623734 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.623907 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260928 22:45:22.624588 11068 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45991
I20260928 22:45:22.624676 11068 heartbeater.cc:464] Registering TS with master...
I20260928 22:45:22.624863 11068 heartbeater.cc:511] Master 127.0.98.62:45991 requested a full tablet report, sending...
I20260928 22:45:22.625232 10608 ts_manager.cc:194] Registered new tserver with Master: 66b0d00006d84abe8e0f83952ef372c1 (127.0.98.3:32769)
I20260928 22:45:22.625579 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41061
I20260928 22:45:22.632552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:45:22.646768 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.633904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.0.98.62:45991)
0928 22:45:22.633984 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.633986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.634030 (+ 44us) server_negotiation.cc:407] Connection header received
0928 22:45:22.634089 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.634090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.634099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.634154 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:45:22.634695 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.635201 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.635907 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.636108 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.636161 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.644303 (+ 8142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:45:22.644304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:45:22.644305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:45:22.644312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:45:22.645602 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.645990 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.645991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.645991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.646042 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:45:22.646216 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:45:22.646217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:45:22.646218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:45:22.646372 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:45:22.646459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.646597 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.646676 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 22:45:22.648461 10608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:45:22.650022 10608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:45:22.656538 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32769 (local address 127.0.0.1:51794)
0928 22:45:22.652292 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.652336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.652402 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.652763 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.652767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.652770 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:22.652848 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.652850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.653924 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.653925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.655062 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.655066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.656251 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.656252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.656336 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.656349 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.656407 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.656456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 22:45:22.657119 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38899 (local address 127.0.0.1:57020)
0928 22:45:22.652769 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.652778 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.652819 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.653146 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.653146 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.653149 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:22.653188 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.653190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.654455 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.654457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.655459 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.655462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.656873 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.656874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.656944 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.656948 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.656995 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.657036 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":67,"threads_started":1}
I20260928 22:45:22.657152 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.98.3:32769)
0928 22:45:22.652559 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.652565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.652576 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:22.652620 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.652622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.652639 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.652873 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:22.653017 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.653826 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.655161 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.656096 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.656163 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.656220 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.657010 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.657068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
I20260928 22:45:22.657354 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.98.2:38899)
0928 22:45:22.652851 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.652855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.652864 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:45:22.652901 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.652903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.652916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.653078 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:22.653269 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.654304 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.655565 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.656690 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.656781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.656839 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.657227 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.657276 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":58,"threads_started":1}
I20260928 22:45:22.658030 11002 tablet_service.cc:1467] Processing CreateTablet for tablet b3a42db80deb477196b4ca77407e0700 (DEFAULT_TABLE table=test-table [id=1bfc16c4e3bc4c39bd037cfcb289309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:22.658062 10868 tablet_service.cc:1467] Processing CreateTablet for tablet b3a42db80deb477196b4ca77407e0700 (DEFAULT_TABLE table=test-table [id=1bfc16c4e3bc4c39bd037cfcb289309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:22.658402 11002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a42db80deb477196b4ca77407e0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:22.658618 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a42db80deb477196b4ca77407e0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:22.660816 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43713 (local address 127.0.0.1:42760)
0928 22:45:22.652627 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.652636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.652697 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.652920 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.652920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.652922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:22.652960 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.652961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.654242 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.654243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.656697 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.656700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.660501 (+ 3801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.660502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.660580 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.660586 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.660618 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.660666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260928 22:45:22.661273 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.652343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.98.1:43713)
0928 22:45:22.652581 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.652585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.652693 (+ 108us) server_negotiation.cc:407] Connection header received
0928 22:45:22.652798 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.652801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.652816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.652924 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:22.653021 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.654132 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.659471 (+ 5339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.660277 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.661009 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.661095 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.661130 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.661187 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260928 22:45:22.661944 10731 tablet_service.cc:1467] Processing CreateTablet for tablet b3a42db80deb477196b4ca77407e0700 (DEFAULT_TABLE table=test-table [id=1bfc16c4e3bc4c39bd037cfcb289309b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:45:22.662272 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a42db80deb477196b4ca77407e0700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:45:22.662739 11086 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Bootstrap starting.
I20260928 22:45:22.663419 11086 tablet_bootstrap.cc:654] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:22.664166 11086 log.cc:824] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:22.664996 11086 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: No bootstrap required, opened a new log
I20260928 22:45:22.665074 11086 ts_tablet_manager.cc:1397] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:45:22.665385 11085 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Bootstrap starting.
I20260928 22:45:22.666299 11085 tablet_bootstrap.cc:654] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:22.666847 11085 log.cc:824] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:22.667447 11086 raft_consensus.cc:348] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.667620 11086 raft_consensus.cc:374] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:22.667649 11086 raft_consensus.cc:729] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cfee0f4aa6487684b2fa8eba9cf544, State: Initialized, Role: FOLLOWER
I20260928 22:45:22.667765 11086 consensus_queue.cc:260] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.668165 11086 ts_tablet_manager.cc:1428] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:45:22.668313 10934 heartbeater.cc:503] Master 127.0.98.62:45991 was elected leader, sending a full tablet report...
I20260928 22:45:22.668459 11088 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Bootstrap starting.
I20260928 22:45:22.669149 11088 tablet_bootstrap.cc:654] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Neither blocks nor log segments found. Creating new log.
I20260928 22:45:22.669847 11088 log.cc:824] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Log is configured to *not* fsync() on all Append() calls
I20260928 22:45:22.670720 11088 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: No bootstrap required, opened a new log
I20260928 22:45:22.670794 11088 ts_tablet_manager.cc:1397] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:45:22.672894 11088 raft_consensus.cc:348] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.673111 11088 raft_consensus.cc:374] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:22.673144 11085 tablet_bootstrap.cc:492] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: No bootstrap required, opened a new log
I20260928 22:45:22.673157 11088 raft_consensus.cc:729] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea702edf30e4c1ba1b6d55d808f0243, State: Initialized, Role: FOLLOWER
I20260928 22:45:22.673220 11085 ts_tablet_manager.cc:1397] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 22:45:22.673280 11088 consensus_queue.cc:260] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.673705 11088 ts_tablet_manager.cc:1428] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 22:45:22.673834 10800 heartbeater.cc:503] Master 127.0.98.62:45991 was elected leader, sending a full tablet report...
I20260928 22:45:22.675161 11085 raft_consensus.cc:348] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.675339 11085 raft_consensus.cc:374] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:45:22.675369 11085 raft_consensus.cc:729] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b0d00006d84abe8e0f83952ef372c1, State: Initialized, Role: FOLLOWER
I20260928 22:45:22.675482 11085 consensus_queue.cc:260] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.675736 11085 ts_tablet_manager.cc:1428] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:45:22.675786 11068 heartbeater.cc:503] Master 127.0.98.62:45991 was elected leader, sending a full tablet report...
W20260928 22:45:22.714231 10935 tablet.cc:2369] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:45:22.821280 10801 tablet.cc:2369] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:45:22.865381 11069 tablet.cc:2369] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:45:22.937215 11092 raft_consensus.cc:482] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:45:22.937345 11092 raft_consensus.cc:504] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.937641 11092 leader_election.cc:288] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66b0d00006d84abe8e0f83952ef372c1 (127.0.98.3:32769), f8cfee0f4aa6487684b2fa8eba9cf544 (127.0.98.2:38899)
I20260928 22:45:22.940644 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.937942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35829 (local address 127.0.98.3:32769)
0928 22:45:22.938003 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.938004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.938015 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:45:22.938098 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.938100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.938115 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.938170 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:22.938332 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.938826 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.939958 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.940326 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.940386 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.940398 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.940533 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.940571 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:45:22.940650 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.938085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41927 (local address 127.0.98.2:38899)
0928 22:45:22.938128 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.938131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.938138 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:45:22.938197 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.938198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.938210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.938270 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:45:22.938417 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.938866 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.939952 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.940331 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.940391 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.940402 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.940553 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.940592 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:45:22.940661 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.937817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32769 (local address 127.0.98.1:35829)
0928 22:45:22.937930 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.937940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.938016 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.938183 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.938184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.938186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:22.938263 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.938265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.938901 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.938902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.939864 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.939867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.940405 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.940406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.940469 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.940473 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.940513 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.940555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":69,"thread_start_us":61,"threads_started":1}
I20260928 22:45:22.940667 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.937982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38899 (local address 127.0.98.1:41927)
0928 22:45:22.938086 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.938095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:45:22.938143 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:45:22.938276 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:45:22.938277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:45:22.938279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:45:22.938352 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.938354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.938941 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.938942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.939855 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:45:22.939859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.940407 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.940408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:45:22.940479 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.940483 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:45:22.940540 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:45:22.940577 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":2,"thread_start_us":36,"threads_started":1}
I20260928 22:45:22.941032 10888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a42db80deb477196b4ca77407e0700" candidate_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" is_pre_election: true
I20260928 22:45:22.941032 11022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a42db80deb477196b4ca77407e0700" candidate_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b0d00006d84abe8e0f83952ef372c1" is_pre_election: true
I20260928 22:45:22.941196 11022 raft_consensus.cc:2454] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea702edf30e4c1ba1b6d55d808f0243 in term 0.
I20260928 22:45:22.941196 10888 raft_consensus.cc:2454] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea702edf30e4c1ba1b6d55d808f0243 in term 0.
I20260928 22:45:22.941437 10689 leader_election.cc:302] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea702edf30e4c1ba1b6d55d808f0243, f8cfee0f4aa6487684b2fa8eba9cf544; no voters:
I20260928 22:45:22.941565 11092 raft_consensus.cc:2781] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:45:22.941618 11092 raft_consensus.cc:482] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:45:22.941650 11092 raft_consensus.cc:3037] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:22.942310 11092 raft_consensus.cc:504] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.942417 11092 leader_election.cc:288] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [CANDIDATE]: Term 1 election: Requested vote from peers 66b0d00006d84abe8e0f83952ef372c1 (127.0.98.3:32769), f8cfee0f4aa6487684b2fa8eba9cf544 (127.0.98.2:38899)
I20260928 22:45:22.942585 11022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a42db80deb477196b4ca77407e0700" candidate_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b0d00006d84abe8e0f83952ef372c1"
I20260928 22:45:22.942615 10888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a42db80deb477196b4ca77407e0700" candidate_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544"
I20260928 22:45:22.942667 11022 raft_consensus.cc:3037] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:22.942673 10888 raft_consensus.cc:3037] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:45:22.943351 10888 raft_consensus.cc:2454] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea702edf30e4c1ba1b6d55d808f0243 in term 1.
I20260928 22:45:22.943468 11022 raft_consensus.cc:2454] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea702edf30e4c1ba1b6d55d808f0243 in term 1.
I20260928 22:45:22.943498 10689 leader_election.cc:302] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea702edf30e4c1ba1b6d55d808f0243, f8cfee0f4aa6487684b2fa8eba9cf544; no voters:
I20260928 22:45:22.943571 11092 raft_consensus.cc:2781] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:45:22.943697 11092 raft_consensus.cc:686] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea702edf30e4c1ba1b6d55d808f0243, State: Running, Role: LEADER
I20260928 22:45:22.943805 11092 consensus_queue.cc:237] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } }
I20260928 22:45:22.944590 10609 catalog_manager.cc:5495] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ea702edf30e4c1ba1b6d55d808f0243 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea702edf30e4c1ba1b6d55d808f0243" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20260928 22:45:22.949647 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.947618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.1:43713)
0928 22:45:22.947660 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:45:22.947662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:45:22.947670 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:45:22.947699 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:45:22.947700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:45:22.947715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:45:22.947771 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:45:22.947905 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.948349 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.949108 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:45:22.949285 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:45:22.949327 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:45:22.949423 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:45:22.949513 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:45:22.949568 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:45:22.949581 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 22:45:22.949596 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:45:22.951161 10888 raft_consensus.cc:1261] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 1 FOLLOWER]: Refusing update from remote peer 1ea702edf30e4c1ba1b6d55d808f0243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:22.951174 11022 raft_consensus.cc:1261] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 1 FOLLOWER]: Refusing update from remote peer 1ea702edf30e4c1ba1b6d55d808f0243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:45:22.951440 11092 consensus_queue.cc:1035] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b0d00006d84abe8e0f83952ef372c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:22.951572 11096 consensus_queue.cc:1035] T b3a42db80deb477196b4ca77407e0700 P 1ea702edf30e4c1ba1b6d55d808f0243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cfee0f4aa6487684b2fa8eba9cf544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:45:22.953877 11101 mvcc.cc:205] Tried to move back new op lower bound from 7334443102005858304 to 7334443101978468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334443102005858304}]
I20260928 22:45:22.954669 11099 mvcc.cc:205] Tried to move back new op lower bound from 7334443102005858304 to 7334443101978468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334443102005858304 or (T in {7334443102005858304})}]
I20260928 22:45:22.954769 11100 mvcc.cc:205] Tried to move back new op lower bound from 7334443102005858304 to 7334443101978468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334443102005858304 or (T in {7334443102005858304})}]
I20260928 22:45:22.955891 10608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:22.955967 10608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:45:22.956596 10608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f43369d2a434b1a84c1b16599227e38: Sending DeleteTablet for 3 replicas of tablet b3a42db80deb477196b4ca77407e0700
I20260928 22:45:22.956970 11002 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a42db80deb477196b4ca77407e0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51794
I20260928 22:45:22.957008 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a42db80deb477196b4ca77407e0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57020
I20260928 22:45:22.957207 11109 tablet_replica.cc:330] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: stopping tablet replica
I20260928 22:45:22.957336 11109 raft_consensus.cc:2229] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:22.957370 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a42db80deb477196b4ca77407e0700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:45:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42760
I20260928 22:45:22.957451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10671
I20260928 22:45:22.957566 11109 raft_consensus.cc:2258] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:22.958315 11109 ts_tablet_manager.cc:1905] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:22.959807 11110 tablet_replica.cc:330] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: stopping tablet replica
I20260928 22:45:22.959911 11110 raft_consensus.cc:2229] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:45:22.959906 11109 ts_tablet_manager.cc:1918] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:22.959970 11109 log.cc:1196] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/b3a42db80deb477196b4ca77407e0700
I20260928 22:45:22.960009 11110 raft_consensus.cc:2258] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:45:22.960258 11109 ts_tablet_manager.cc:1939] T b3a42db80deb477196b4ca77407e0700 P 66b0d00006d84abe8e0f83952ef372c1: Deleting consensus metadata
I20260928 22:45:22.960412 11110 ts_tablet_manager.cc:1905] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:45:22.960636 10595 catalog_manager.cc:4841] TS 66b0d00006d84abe8e0f83952ef372c1 (127.0.98.3:32769): tablet b3a42db80deb477196b4ca77407e0700 (table test-table [id=1bfc16c4e3bc4c39bd037cfcb289309b]) successfully deleted
I20260928 22:45:22.961855 11110 ts_tablet_manager.cc:1918] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:45:22.961921 11110 log.cc:1196] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/b3a42db80deb477196b4ca77407e0700
I20260928 22:45:22.962129 11110 ts_tablet_manager.cc:1939] T b3a42db80deb477196b4ca77407e0700 P f8cfee0f4aa6487684b2fa8eba9cf544: Deleting consensus metadata
I20260928 22:45:22.962503 10595 catalog_manager.cc:4841] TS f8cfee0f4aa6487684b2fa8eba9cf544 (127.0.98.2:38899): tablet b3a42db80deb477196b4ca77407e0700 (table test-table [id=1bfc16c4e3bc4c39bd037cfcb289309b]) successfully deleted
W20260928 22:45:22.963866 10594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43713 (error 108)
I20260928 22:45:22.964089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10805
I20260928 22:45:22.965129 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:45:22.964266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43713 (local address 127.0.0.1:42776)
0928 22:45:22.965053 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 22:45:22.965097 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745}
W20260928 22:45:22.965247 10594 catalog_manager.cc:4584] TS 1ea702edf30e4c1ba1b6d55d808f0243 (127.0.98.1:43713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3a42db80deb477196b4ca77407e0700: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43713: connect: Connection refused (error 111)
I20260928 22:45:22.969008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10939
I20260928 22:45:22.973480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 10577
2026-09-28T22:45:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2047 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:45:23 dist-test-slave-pxr1 krb5kdc[11115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:45:23 dist-test-slave-pxr1 krb5kdc[11115](info): set up 2 sockets
Sep 28 22:45:23 dist-test-slave-pxr1 krb5kdc[11115](info): commencing operation
krb5kdc: starting...
W20260928 22:45:23.917446 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:45:23 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:45:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 22:45:24.071493 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskntG_DW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/postgres -l logfile start
2026-09-28 22:45:26.396 UTC [11145] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:45:26.396 UTC [11145] LOG: listening on IPv4 address "127.0.98.20", port 59989
2026-09-28 22:45:26.399 UTC [11145] LOG: listening on Unix socket "/tmp/.s.PGSQL.59989"
2026-09-28 22:45:26.426 UTC [11148] LOG: database system was shut down at 2026-09-28 22:45:25 UTC
2026-09-28 22:45:26.429 UTC [11145] LOG: database system is ready to accept connections
I20260928 22:45:27.289335 392 mini_postgres.cc:96] Postgres bound to 59989
2026-09-28 22:45:27.294 UTC [11156] FATAL: database "slave" does not exist
127.0.98.20:59989 - accepting connections
I20260928 22:45:27.295491 392 mini_ranger.cc:161] Starting Ranger...
I20260928 22:45:27.309299 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:45:27.756920 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:45:27.757002 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin
2026-09-28 22:45:28,015 [I] DB FLAVOR :POSTGRES
2026-09-28 22:45:28,015 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:45:28,015 [I] Checking connection..
2026-09-28 22:45:28,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:45:28,252 [I] Checking connection passed.
2026-09-28 22:45:28,253 [I] --------- Verifying version history table ---------
2026-09-28 22:45:28,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:45:28,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:45:28,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:45:28,757 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:45:28,758 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:45:28,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:45:29.001 UTC [11220] WARNING: there is no transaction in progress
2026-09-28 22:45:29,007 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:45:29,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:45:29,251 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:45:29,252 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:45:29,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:45:29,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:45:29,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pxr1.c.gcp-upstream.internal','N') ;"
2026-09-28 22:45:29,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:45:30,234 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:45:30,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:45:30,482 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:45:30,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:45:30,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:45:30,969 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:45:30,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:45:31.902 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:31.918 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:31.935 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:31.950 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:31.965 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:31.982 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.038 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.045 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.059 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.069 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.079 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.093 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.103 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.109 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.115 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32.121 UTC [11328] WARNING: there is no transaction in progress
2026-09-28 22:45:32,743 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:45:32,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:45:32,962 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:45:32,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:45:33,218 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:45:33,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:45:33,471 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:45:33,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:45:33,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:45:33,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pxr1.c.gcp-upstream.internal';"
2026-09-28 22:45:34,205 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:45:34,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:45:34,462 [I] DB_PATCHES have already been applied
I20260928 22:45:34.467440 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskntG_DW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:45:34.467504 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 22:45:34.470307 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36339
Sep 28, 2026 10:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:45:35.468901 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.173s user 0.003s sys 0.005s
Sep 28, 2026 10:45:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:45:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:45:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:45:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:45:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:45:55.820308 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:45:55 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635555, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:45:55.897960 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:45:55.907078 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:45:56.721184 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:45:56.721269 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms
I20260928 22:45:56.721285 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59989
2026-09-28 22:45:56,890 [I] DB FLAVOR :POSTGRES
2026-09-28 22:45:56,890 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:45:56,890 [I] Checking connection
2026-09-28 22:45:56,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:45:57,129 [I] connection success
2026-09-28 22:45:57,129 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:45:57,130 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:45:57,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:45:57,377 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:45:57,377 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:45:57,377 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:45:57,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59989/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:45:57,639 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:45:58.098807 392 mini_ranger_kms.cc:310] Created kms service
I20260928 22:45:58.286795 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:45:58.331946 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:45:58 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635555, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:45:58.746399 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:45:58.746531 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:45:58.746551 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 22:45:58.749763 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11508
I20260928 22:45:58.749897 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49597
Sep 28, 2026 10:45:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:45:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:45:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:45:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:46:00.864495 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.967s user 0.005s sys 0.148s
I20260928 22:46:00.864609 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:46:00.864679 392 mini_ranger_kms.cc:401] 127.0.98.20:49597/kms/v1/keys
Sep 28 22:46:00 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:46:01 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 22:46:01 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635560, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 22:46:04 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:46:04.263814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49597/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:04.344764 11559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:04.344985 11559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:04.345007 11559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:04.346235 11559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:04.346283 11559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:04.346297 11559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:04.346307 11559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:04.346318 11559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:04.347945 11559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49597/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:04.348171 11559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:04.348384 11559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:04.353298 11565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:04.353331 11564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:04.353433 11567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:04.353410 11559 server_base.cc:1034] running on GCE node
I20260928 22:46:04.354692 11559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:04.354991 11559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:04.356156 11559 hybrid_clock.cc:648] HybridClock initialized: now 1790635564356141 us; error 35 us; skew 500 ppm
Sep 28 22:46:04 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:04.358735 11559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:04.359216 11559 webserver.cc:460] Webserver started at http://127.0.98.62:36403/ using document root <none> and password file <none>
I20260928 22:46:04.359441 11559 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:04.359480 11559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:04.359614 11559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:04 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:04.415822 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "1dbf36d8e730466f8e29a3b86217e276"
format_stamp: "Formatted at 2026-09-28 22:46:04 on dist-test-slave-pxr1"
server_key: "735476447858bc6fc4e933c9a77dd940"
server_key_iv: "f48995e97ff6fbb20adc6f7c2faf75cb"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:04.416199 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dbf36d8e730466f8e29a3b86217e276"
format_stamp: "Formatted at 2026-09-28 22:46:04 on dist-test-slave-pxr1"
server_key: "735476447858bc6fc4e933c9a77dd940"
server_key_iv: "f48995e97ff6fbb20adc6f7c2faf75cb"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:04.417608 11559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:46:04.433319 11574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:04.433566 11559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:46:04.433665 11559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "1dbf36d8e730466f8e29a3b86217e276"
format_stamp: "Formatted at 2026-09-28 22:46:04 on dist-test-slave-pxr1"
server_key: "735476447858bc6fc4e933c9a77dd940"
server_key_iv: "f48995e97ff6fbb20adc6f7c2faf75cb"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:04.433768 11559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:04.497498 11559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:04.499514 11559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:04.499671 11559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:04.504689 11559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260928 22:46:04.504745 11626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260928 22:46:04.505136 11559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:04.505779 11627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:04.508034 11627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: Bootstrap starting.
I20260928 22:46:04.508651 11627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:04.508935 11627 log.cc:824] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:04.509596 11627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: No bootstrap required, opened a new log
I20260928 22:46:04.510082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 11559
I20260928 22:46:04.510205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:46:04.510406 392 external_mini_cluster.cc:1428] Setting key 597e5c6e52729645eec319e38d57f36a
I20260928 22:46:04.511855 11627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260928 22:46:04.512003 11627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:04.512025 11627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf36d8e730466f8e29a3b86217e276, State: Initialized, Role: FOLLOWER
I20260928 22:46:04.512140 11627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260928 22:46:04.512210 11627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:04.512262 11627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:04.512339 11627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:04.513038 11627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260928 22:46:04.513142 11627 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbf36d8e730466f8e29a3b86217e276; no voters:
I20260928 22:46:04.513332 11627 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:46:04.513585 11633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:04.513669 11627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:04.513784 11633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf36d8e730466f8e29a3b86217e276, State: Running, Role: LEADER
I20260928 22:46:04.513962 11633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260928 22:46:04.514379 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbf36d8e730466f8e29a3b86217e276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260928 22:46:04.514521 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:04.514693 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbf36d8e730466f8e29a3b86217e276. Latest consensus state: current_term: 1 leader_uuid: "1dbf36d8e730466f8e29a3b86217e276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbf36d8e730466f8e29a3b86217e276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260928 22:46:04.514802 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:04.515048 11627 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:46:04.515319 11642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:04.520130 11642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:46:04 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:04.524286 11642 catalog_manager.cc:1347] Generated new cluster ID: 16655993eccd402096b33dda29021318
I20260928 22:46:04.524338 11642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:04.527724 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:04.510949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.98.62:42343)
0928 22:46:04.511153 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:04.511163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:04.512307 (+ 1144us) server_negotiation.cc:407] Connection header received
0928 22:46:04.512438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:04.512446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:04.512505 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:04.512649 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:04.513395 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:04.514804 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:04.515820 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:04.519553 (+ 3733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:04.519748 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:04.522865 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:04.522870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:04.522871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:04.522884 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:04.525434 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:04.525874 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:04.525878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:04.525878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:04.525988 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:04.526218 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:04.526220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:04.526220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:04.526601 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:04.527236 (+ 635us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:04.527312 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:46:04.527460 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260928 22:46:04.538633 11642 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:46:04.539448 11642 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:04.544840 11642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: Generated new TSK 0
I20260928 22:46:04.545140 11642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:46:05.361332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:05.488854 11672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:05.489235 11672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:05.489303 11672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:05.491156 11672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:05.491282 11672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:05.491361 11672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:46:05.493637 11672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:05.493918 11672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:05.494210 11672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:05.500394 11678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:05.500459 11677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:05.500409 11680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:05.500759 11672 server_base.cc:1034] running on GCE node
I20260928 22:46:05.501053 11672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:05.501346 11672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:05.502811 11672 hybrid_clock.cc:648] HybridClock initialized: now 1790635565502782 us; error 40 us; skew 500 ppm
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:05.506275 11672 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:46:05.506866 11672 webserver.cc:460] Webserver started at http://127.0.98.1:43519/ using document root <none> and password file <none>
I20260928 22:46:05.507174 11672 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:05.507277 11672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:05.507458 11672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:05.522328 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e45816b075542e9852c6bc77dd7078c"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "aca7bf990df1f126b4206853c5e27ab2"
server_key_iv: "d380be8bfcd7ecc285ce910ac1fe3ded"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.522924 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e45816b075542e9852c6bc77dd7078c"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "aca7bf990df1f126b4206853c5e27ab2"
server_key_iv: "d380be8bfcd7ecc285ce910ac1fe3ded"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.524922 11672 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:05.540896 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.541263 11672 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 22:46:05.541457 11672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "8e45816b075542e9852c6bc77dd7078c"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "aca7bf990df1f126b4206853c5e27ab2"
server_key_iv: "d380be8bfcd7ecc285ce910ac1fe3ded"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.541695 11672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:05.562552 11672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:05.565153 11672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:05.565299 11672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:05.565496 11672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:05.565886 11672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:05.565929 11672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.565966 11672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:05.566009 11672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.573323 11672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46705
I20260928 22:46:05.573525 11802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46705 every 8 connection(s)
I20260928 22:46:05.573801 11672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:46:05.580397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 11672
I20260928 22:46:05.580499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:46:05.580681 392 external_mini_cluster.cc:1428] Setting key 868d95b327dbdb0c9e0a4279efc85098
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:05.590227 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:05.574734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:60345)
0928 22:46:05.574913 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:05.574964 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:05.575606 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:05.575889 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:05.575894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:05.576152 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:05.576454 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:05.576462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.577463 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.577464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:05.578764 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:05.578767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.579262 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.579263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:05.579433 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:05.580094 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:05.580105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:05.582435 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:05.588408 (+ 5973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:05.588421 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:05.588424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:05.588812 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:05.589049 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:05.589050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:05.589050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:05.589194 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:05.589519 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:05.589526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:05.589721 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 22:46:05.589813 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:46:05.589979 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 22:46:05.591065 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:05.574904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60345 (local address 127.0.98.62:42343)
0928 22:46:05.575020 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:05.575022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:05.575040 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:46:05.575723 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:05.575724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:05.575738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:05.575811 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:05.576561 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.577362 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.578860 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.579166 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.579235 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:05.586306 (+ 7071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:05.586308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:05.586309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:05.586319 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:05.588265 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:05.588913 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:05.588914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:05.588915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:05.588980 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:05.589273 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:05.589274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:05.589274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:05.589443 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:05.590831 (+ 1388us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:05.590877 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 22:46:05.590966 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260928 22:46:05.591758 11803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260928 22:46:05.591859 11803 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:05.592051 11803 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260928 22:46:05.592760 11591 ts_manager.cc:194] Registered new tserver with Master: 8e45816b075542e9852c6bc77dd7078c (127.0.98.1:46705)
I20260928 22:46:05.593766 11591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60345
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:46:05.637053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 22:46:05.763751 11809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:05.764181 11809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:05.764268 11809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:05.766199 11809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:05.766453 11809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:05.766564 11809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:46:05.769017 11809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:05.769500 11809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:05.769833 11809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:05.776964 11817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:05.776975 11819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:05.777390 11816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:05.778090 11809 server_base.cc:1034] running on GCE node
I20260928 22:46:05.778367 11809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:05.778644 11809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:05.781922 11809 hybrid_clock.cc:648] HybridClock initialized: now 1790635565781889 us; error 40 us; skew 500 ppm
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:05.785423 11809 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:46:05.785985 11809 webserver.cc:460] Webserver started at http://127.0.98.2:40649/ using document root <none> and password file <none>
I20260928 22:46:05.786227 11809 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:05.786278 11809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:05.786405 11809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:05.801472 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "265d5c6358854b128cbe6d723275dda2"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "a3bc06ca306d9244557124b656a0744b"
server_key_iv: "283fe19dafabc1b92f68bd45fd64bae8"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.802057 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "265d5c6358854b128cbe6d723275dda2"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "a3bc06ca306d9244557124b656a0744b"
server_key_iv: "283fe19dafabc1b92f68bd45fd64bae8"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.804808 11809 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:05.818511 11826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.818790 11809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:05.818882 11809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "265d5c6358854b128cbe6d723275dda2"
format_stamp: "Formatted at 2026-09-28 22:46:05 on dist-test-slave-pxr1"
server_key: "a3bc06ca306d9244557124b656a0744b"
server_key_iv: "283fe19dafabc1b92f68bd45fd64bae8"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:05.818976 11809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:05.827998 11809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:05.829873 11809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:05.830004 11809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:05.830224 11809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:05.830562 11809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:05.830644 11809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.830713 11809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:05.830765 11809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:05.838234 11809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36505
I20260928 22:46:05.838331 11940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36505 every 8 connection(s)
I20260928 22:46:05.838768 11809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:46:05.840013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 11809
I20260928 22:46:05.840123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:46:05.840305 392 external_mini_cluster.cc:1428] Setting key 89962ce01a47b86e7f5b0e9c7c8a5e61
Sep 28 22:46:05 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:05.850158 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:05.839554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:50007)
0928 22:46:05.839719 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:05.839769 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:05.840449 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:05.840807 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:05.840811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:05.841062 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:05.841361 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:05.841368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.842347 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.842348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:05.843653 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:05.843657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.844180 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.844181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:05.844325 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:05.844965 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:05.844975 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:05.846568 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:05.848407 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:05.848416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:05.848420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:05.848791 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:05.849005 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:05.849006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:05.849006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:05.849145 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:05.849442 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:05.849449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:05.849643 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 22:46:05.849743 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:46:05.849912 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 22:46:05.851064 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:05.839570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50007 (local address 127.0.98.62:42343)
0928 22:46:05.839625 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:05.839627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:05.839866 (+ 239us) server_negotiation.cc:407] Connection header received
0928 22:46:05.840612 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:05.840614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:05.840627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:05.840696 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:05.841479 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.842227 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.843759 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:05.844077 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:05.844151 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:05.846684 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:05.846686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:05.846686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:05.846695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:05.848307 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:05.848877 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:05.848878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:05.848879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:05.848939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:05.849212 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:05.849213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:05.849213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:05.849353 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:05.849406 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:05.850871 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0928 22:46:05.850969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:46:05.851673 11941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260928 22:46:05.851784 11941 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:05.851982 11941 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260928 22:46:05.852451 11591 ts_manager.cc:194] Registered new tserver with Master: 265d5c6358854b128cbe6d723275dda2 (127.0.98.2:36505)
I20260928 22:46:05.852977 11591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50007
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:46:05.877836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:06.015820 11946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:06.016196 11946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:06.016278 11946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:06.018356 11946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:06.018468 11946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:06.018567 11946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:46:06.020937 11946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:06.021421 11946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:06.021734 11946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:06.028316 11952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:06.028731 11951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:06.028980 11954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:06.029798 11946 server_base.cc:1034] running on GCE node
I20260928 22:46:06.030128 11946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:06.030401 11946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:06.031564 11946 hybrid_clock.cc:648] HybridClock initialized: now 1790635566031545 us; error 31 us; skew 500 ppm
Sep 28 22:46:06 dist-test-slave-pxr1 krb5kdc[11115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:06.034754 11946 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:46:06.035279 11946 webserver.cc:460] Webserver started at http://127.0.98.3:39899/ using document root <none> and password file <none>
I20260928 22:46:06.035543 11946 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:06.035595 11946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:06.035722 11946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:06 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:06.049019 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb81d5b5397f44229bc4295505041809"
format_stamp: "Formatted at 2026-09-28 22:46:06 on dist-test-slave-pxr1"
server_key: "951b27c475c292bbe1b41e134d354356"
server_key_iv: "d7da9a71bbb8520712d5177e6a2ab32c"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:06.049561 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb81d5b5397f44229bc4295505041809"
format_stamp: "Formatted at 2026-09-28 22:46:06 on dist-test-slave-pxr1"
server_key: "951b27c475c292bbe1b41e134d354356"
server_key_iv: "d7da9a71bbb8520712d5177e6a2ab32c"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:06.051368 11946 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:06.064679 11961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:06.064934 11946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:06.065034 11946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "eb81d5b5397f44229bc4295505041809"
format_stamp: "Formatted at 2026-09-28 22:46:06 on dist-test-slave-pxr1"
server_key: "951b27c475c292bbe1b41e134d354356"
server_key_iv: "d7da9a71bbb8520712d5177e6a2ab32c"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:06.065131 11946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:06.085436 11946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:06.087792 11946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:06.087994 11946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:06.088271 11946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:06.088681 11946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:06.088797 11946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:06.088869 11946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:06.088924 11946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:06.094930 11946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43109
I20260928 22:46:06.095412 11946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:46:06.099216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 11946
I20260928 22:46:06.099327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:46:06.099509 392 external_mini_cluster.cc:1428] Setting key bf310dee5fe8b891cb9e3439671f697c
I20260928 22:46:06.106647 12074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43109 every 8 connection(s)
Sep 28 22:46:06 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:06.119796 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.107803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44761 (local address 127.0.98.62:42343)
0928 22:46:06.107847 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.107849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.108199 (+ 350us) server_negotiation.cc:407] Connection header received
0928 22:46:06.108925 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.108926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.108965 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.108996 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:06.110418 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.111226 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.113201 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.113492 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.113541 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.116375 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:06.116376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:06.116377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:06.116385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:06.118109 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:06.118626 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:06.118627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:06.118628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:06.118691 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:06.119005 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:06.119006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:06.119006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:06.119149 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:06.119193 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.119601 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.119696 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:46:06.120693 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.107476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:44761)
0928 22:46:06.108061 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.108110 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.108799 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.109193 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.109198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.109458 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:06.110296 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.110304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.111775 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.111776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.113097 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.113100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.113639 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.113640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.113832 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.114480 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:06.114490 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:06.116256 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:06.118212 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:06.118219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:06.118222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:06.118529 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:06.118765 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:06.118766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:06.118766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:06.118913 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:06.119241 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:06.119248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:06.119433 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.120183 (+ 750us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.120363 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":114,"threads_started":1}
I20260928 22:46:06.121435 12075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260928 22:46:06.121536 12075 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:06.121730 12075 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260928 22:46:06.122699 11591 ts_manager.cc:194] Registered new tserver with Master: eb81d5b5397f44229bc4295505041809 (127.0.98.3:43109)
I20260928 22:46:06.123209 11591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44761
I20260928 22:46:06.133348 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:46:06.147378 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.137457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.98.62:42343)
0928 22:46:06.137511 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.137513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.138101 (+ 588us) server_negotiation.cc:407] Connection header received
0928 22:46:06.138217 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.138218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.138231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.138260 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:06.139058 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.139829 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.141071 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.141375 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.141422 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.144199 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:06.144201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:06.144202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:06.144212 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:06.145923 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:06.146381 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:06.146382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:06.146383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:06.146446 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:06.146660 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:06.146660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:06.146661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:06.146804 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:06.146883 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.147193 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.147278 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:46:06.149724 11591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:46:06.152105 11591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:46:06.170825 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.156097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46705 (local address 127.0.0.1:41476)
0928 22:46:06.157067 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.157075 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.157112 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.157906 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.157909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.157912 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:06.158008 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.158010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.164692 (+ 6682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.164693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.165790 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.165794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.170481 (+ 4687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.170482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.170597 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.170617 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.170669 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.170729 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":57,"threads_started":1}
I20260928 22:46:06.171967 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.156253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43109 (local address 127.0.0.1:41292)
0928 22:46:06.157175 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.157183 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.157214 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.158084 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.158085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.158087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:06.158120 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.158122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.162889 (+ 4767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.162890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.164584 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.164589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.171716 (+ 7127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.171717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.171788 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.171793 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.171825 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.171871 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":59,"threads_started":1}
I20260928 22:46:06.172580 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.156192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.1:46705)
0928 22:46:06.157555 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.157559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.157567 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:06.157597 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.157599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.157614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.158192 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:06.158259 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.162783 (+ 4524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.166918 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.169611 (+ 2693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.172324 (+ 2713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.172421 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.172452 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.172504 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":57,"threads_started":1}
I20260928 22:46:06.173170 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.156461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.0.98.3:43109)
0928 22:46:06.157743 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.157747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.157755 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:06.157787 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.157789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.157803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.158726 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:06.158809 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.159887 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.165931 (+ 6044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.166795 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.172874 (+ 6079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.173006 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.173038 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.173089 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":86,"threads_started":1}
I20260928 22:46:06.173645 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.155837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36505 (local address 127.0.0.1:52408)
0928 22:46:06.156842 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.156888 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.156967 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.160015 (+ 3048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.160016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.160018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:06.160062 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.160063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.167323 (+ 7260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.167325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.168407 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.168411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.173404 (+ 4993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.173405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.173487 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.173493 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.173528 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.173574 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":73,"threads_started":1}
I20260928 22:46:06.174337 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.155946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.0.98.2:36505)
0928 22:46:06.157351 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.157357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.157367 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:46:06.157411 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.157413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.157429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.160135 (+ 2706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:06.160220 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.162426 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.168543 (+ 6117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.170363 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.174062 (+ 3699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.174178 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.174208 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.174257 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":77,"threads_started":1}
I20260928 22:46:06.175238 11737 tablet_service.cc:1467] Processing CreateTablet for tablet 1224d368eafb45c4b1920150b97f8b32 (DEFAULT_TABLE table=test-table [id=66f488bfcce849c38cde5f97cc86ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:06.175370 12009 tablet_service.cc:1467] Processing CreateTablet for tablet 1224d368eafb45c4b1920150b97f8b32 (DEFAULT_TABLE table=test-table [id=66f488bfcce849c38cde5f97cc86ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:06.175624 11737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1224d368eafb45c4b1920150b97f8b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:06.175688 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1224d368eafb45c4b1920150b97f8b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:06.179992 12095 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Bootstrap starting.
I20260928 22:46:06.180262 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 1224d368eafb45c4b1920150b97f8b32 (DEFAULT_TABLE table=test-table [id=66f488bfcce849c38cde5f97cc86ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:06.180655 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1224d368eafb45c4b1920150b97f8b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:06.181190 12095 tablet_bootstrap.cc:654] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:06.182068 12095 log.cc:824] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:06.182297 12094 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Bootstrap starting.
I20260928 22:46:06.183049 12094 tablet_bootstrap.cc:654] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:06.184258 12094 log.cc:824] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:06.185104 12094 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: No bootstrap required, opened a new log
I20260928 22:46:06.185230 12094 ts_tablet_manager.cc:1397] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:46:06.187889 12094 raft_consensus.cc:348] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.188251 12094 raft_consensus.cc:374] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:06.188349 12094 raft_consensus.cc:729] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e45816b075542e9852c6bc77dd7078c, State: Initialized, Role: FOLLOWER
I20260928 22:46:06.188580 12094 consensus_queue.cc:260] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.189029 12094 ts_tablet_manager.cc:1428] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:46:06.189227 11803 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
I20260928 22:46:06.189683 12097 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Bootstrap starting.
I20260928 22:46:06.190218 12095 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: No bootstrap required, opened a new log
I20260928 22:46:06.190357 12095 ts_tablet_manager.cc:1397] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260928 22:46:06.193033 12095 raft_consensus.cc:348] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.193288 12095 raft_consensus.cc:374] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:06.193379 12095 raft_consensus.cc:729] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb81d5b5397f44229bc4295505041809, State: Initialized, Role: FOLLOWER
I20260928 22:46:06.193590 12095 consensus_queue.cc:260] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.193958 12095 ts_tablet_manager.cc:1428] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 22:46:06.194121 12075 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
I20260928 22:46:06.195998 12097 tablet_bootstrap.cc:654] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:06.196956 12097 log.cc:824] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:06.198980 12097 tablet_bootstrap.cc:492] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: No bootstrap required, opened a new log
I20260928 22:46:06.199115 12097 ts_tablet_manager.cc:1397] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260928 22:46:06.201956 12097 raft_consensus.cc:348] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.202253 12097 raft_consensus.cc:374] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:06.202346 12097 raft_consensus.cc:729] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265d5c6358854b128cbe6d723275dda2, State: Initialized, Role: FOLLOWER
I20260928 22:46:06.202577 12097 consensus_queue.cc:260] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.202963 12097 ts_tablet_manager.cc:1428] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:46:06.203130 11941 heartbeater.cc:503] Master 127.0.98.62:42343 was elected leader, sending a full tablet report...
Sep 28 22:46:06 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260928 22:46:06.330029 11804 tablet.cc:2369] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:46:06.339728 11942 tablet.cc:2369] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:46:06.358242 12076 tablet.cc:2369] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:46:06 dist-test-slave-pxr1 krb5kdc[11115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:06.533707 12100 raft_consensus.cc:482] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:46:06.533867 12100 raft_consensus.cc:504] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.534312 12100 leader_election.cc:288] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 265d5c6358854b128cbe6d723275dda2 (127.0.98.2:36505), 8e45816b075542e9852c6bc77dd7078c (127.0.98.1:46705)
I20260928 22:46:06.539489 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.534568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46873 (local address 127.0.98.2:36505)
0928 22:46:06.534622 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.534624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.534741 (+ 117us) server_negotiation.cc:407] Connection header received
0928 22:46:06.534798 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.534799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.534808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.534863 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:06.535063 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.535750 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.538586 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.539132 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.539201 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.539216 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.539374 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.539414 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:46:06.539489 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.534441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36505 (local address 127.0.98.3:46873)
0928 22:46:06.534672 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.534683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.534741 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.534918 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.534919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.534921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:06.534991 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.534993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.537228 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.537229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.538450 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.538455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.539225 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.539227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.539310 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.539314 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.539358 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.539405 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260928 22:46:06.539961 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1224d368eafb45c4b1920150b97f8b32" candidate_uuid: "eb81d5b5397f44229bc4295505041809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265d5c6358854b128cbe6d723275dda2" is_pre_election: true
I20260928 22:46:06.540148 11895 raft_consensus.cc:2454] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb81d5b5397f44229bc4295505041809 in term 0.
I20260928 22:46:06.540401 11965 leader_election.cc:302] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265d5c6358854b128cbe6d723275dda2, eb81d5b5397f44229bc4295505041809; no voters:
I20260928 22:46:06.540539 12100 raft_consensus.cc:2781] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:46:06.540633 12100 raft_consensus.cc:482] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:46:06.540663 12100 raft_consensus.cc:3037] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:06.541618 12100 raft_consensus.cc:504] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.541790 12100 leader_election.cc:288] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [CANDIDATE]: Term 1 election: Requested vote from peers 265d5c6358854b128cbe6d723275dda2 (127.0.98.2:36505), 8e45816b075542e9852c6bc77dd7078c (127.0.98.1:46705)
I20260928 22:46:06.542136 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1224d368eafb45c4b1920150b97f8b32" candidate_uuid: "eb81d5b5397f44229bc4295505041809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265d5c6358854b128cbe6d723275dda2"
I20260928 22:46:06.542235 11895 raft_consensus.cc:3037] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:06.543085 11895 raft_consensus.cc:2454] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb81d5b5397f44229bc4295505041809 in term 1.
I20260928 22:46:06.543287 11965 leader_election.cc:302] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265d5c6358854b128cbe6d723275dda2, eb81d5b5397f44229bc4295505041809; no voters:
I20260928 22:46:06.543390 12100 raft_consensus.cc:2781] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:06.543569 12100 raft_consensus.cc:686] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 1 LEADER]: Becoming Leader. State: Replica: eb81d5b5397f44229bc4295505041809, State: Running, Role: LEADER
I20260928 22:46:06.543681 12100 consensus_queue.cc:237] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } }
I20260928 22:46:06.543731 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.534884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59647 (local address 127.0.98.1:46705)
0928 22:46:06.534927 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.534929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.539699 (+ 4770us) server_negotiation.cc:407] Connection header received
0928 22:46:06.539731 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.539732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.539745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.539787 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:06.539931 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.540564 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.541537 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.542010 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.542073 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.542085 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.543609 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.543664 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:46:06.544381 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.534519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46705 (local address 127.0.98.3:59647)
0928 22:46:06.539579 (+ 5060us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.539592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.539642 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.539831 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:06.539832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:06.539833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:06.539868 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.539869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.540651 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.540652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.541450 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:06.541453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.543393 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.543394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:06.543461 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.543466 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:46:06.543527 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:46:06.543570 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5047,"thread_start_us":105,"threads_started":1}
I20260928 22:46:06.544562 11591 catalog_manager.cc:5495] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb81d5b5397f44229bc4295505041809 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eb81d5b5397f44229bc4295505041809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb81d5b5397f44229bc4295505041809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43109 } health_report { overall_health: HEALTHY } } }
I20260928 22:46:06.545152 11755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1224d368eafb45c4b1920150b97f8b32" candidate_uuid: "eb81d5b5397f44229bc4295505041809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e45816b075542e9852c6bc77dd7078c" is_pre_election: true
I20260928 22:46:06.545307 11755 raft_consensus.cc:2454] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb81d5b5397f44229bc4295505041809 in term 0.
I20260928 22:46:06.545429 11757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1224d368eafb45c4b1920150b97f8b32" candidate_uuid: "eb81d5b5397f44229bc4295505041809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e45816b075542e9852c6bc77dd7078c"
I20260928 22:46:06.545495 11757 raft_consensus.cc:3037] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:06.546353 11757 raft_consensus.cc:2454] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb81d5b5397f44229bc4295505041809 in term 1.
I20260928 22:46:06.610178 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.606653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.0.98.3:43109)
0928 22:46:06.606696 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:06.606698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:06.606705 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:46:06.606746 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:06.606747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:06.606771 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:06.606832 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:46:06.607060 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.607800 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.609003 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:06.609304 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:06.609369 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:06.609586 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:46:06.609724 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:46:06.609798 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:06.610053 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 22:46:06.610076 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:46:06.624818 11757 raft_consensus.cc:1261] T 1224d368eafb45c4b1920150b97f8b32 P 8e45816b075542e9852c6bc77dd7078c [term 1 FOLLOWER]: Refusing update from remote peer eb81d5b5397f44229bc4295505041809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:06.625310 12100 consensus_queue.cc:1035] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e45816b075542e9852c6bc77dd7078c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:06.627705 11895 raft_consensus.cc:1261] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 1 FOLLOWER]: Refusing update from remote peer eb81d5b5397f44229bc4295505041809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:06.628090 12100 consensus_queue.cc:1035] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265d5c6358854b128cbe6d723275dda2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:06.631641 12109 mvcc.cc:205] Tried to move back new op lower bound from 7334443280892231680 to 7334443280563617792. Current Snapshot: MvccSnapshot[applied={T|T < 7334443280892231680}]
I20260928 22:46:06.632247 12110 mvcc.cc:205] Tried to move back new op lower bound from 7334443280892231680 to 7334443280563617792. Current Snapshot: MvccSnapshot[applied={T|T < 7334443280892231680}]
I20260928 22:46:06.635121 12112 mvcc.cc:205] Tried to move back new op lower bound from 7334443280892231680 to 7334443280563617792. Current Snapshot: MvccSnapshot[applied={T|T < 7334443280892231680}]
I20260928 22:46:06.637887 11591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:06.637991 11591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:06.638679 11591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dbf36d8e730466f8e29a3b86217e276: Sending DeleteTablet for 3 replicas of tablet 1224d368eafb45c4b1920150b97f8b32
I20260928 22:46:06.639189 11874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1224d368eafb45c4b1920150b97f8b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52408
I20260928 22:46:06.639416 11737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1224d368eafb45c4b1920150b97f8b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41476
I20260928 22:46:06.639623 12009 tablet_service.cc:1514] Processing DeleteTablet for tablet 1224d368eafb45c4b1920150b97f8b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41292
I20260928 22:46:06.640311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 11672
I20260928 22:46:06.648229 12121 tablet_replica.cc:330] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: stopping tablet replica
I20260928 22:46:06.648469 12119 tablet_replica.cc:330] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: stopping tablet replica
I20260928 22:46:06.648706 12119 raft_consensus.cc:2229] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:46:06.648909 12121 raft_consensus.cc:2229] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:46:06.649121 12121 raft_consensus.cc:2258] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:06.649310 12119 raft_consensus.cc:2258] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:06.650058 12119 ts_tablet_manager.cc:1905] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:46:06.651912 11576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46705 (error 108)
I20260928 22:46:06.652264 12119 ts_tablet_manager.cc:1918] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:46:06.652424 12119 log.cc:1196] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/1224d368eafb45c4b1920150b97f8b32
I20260928 22:46:06.652748 12119 ts_tablet_manager.cc:1939] T 1224d368eafb45c4b1920150b97f8b32 P 265d5c6358854b128cbe6d723275dda2: Deleting consensus metadata
I20260928 22:46:06.653016 12121 ts_tablet_manager.cc:1905] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:46:06.653246 11578 catalog_manager.cc:4841] TS 265d5c6358854b128cbe6d723275dda2 (127.0.98.2:36505): tablet 1224d368eafb45c4b1920150b97f8b32 (table test-table [id=66f488bfcce849c38cde5f97cc86ef01]) successfully deleted
I20260928 22:46:06.654068 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:06.653944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46705 (local address 127.0.0.1:41492)
0928 22:46:06.653997 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.654033 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:46:06.654275 11576 catalog_manager.cc:4584] TS 8e45816b075542e9852c6bc77dd7078c (127.0.98.1:46705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1224d368eafb45c4b1920150b97f8b32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46705: connect: Connection refused (error 111)
I20260928 22:46:06.654518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 11809
I20260928 22:46:06.660030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 11946
I20260928 22:46:06.660080 12121 ts_tablet_manager.cc:1918] T 1224d368eafb45c4b1920150b97f8b32 P eb81d5b5397f44229bc4295505041809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 22:46:06.666530 12092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:46:06.666232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43109 (local address 127.0.0.1:41314)
0928 22:46:06.666294 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:06.666305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:06.666346 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:06.666467 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:46:06.667119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 11559
I20260928 22:46:06.673303 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:46:07.007964 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:46:07.008069 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:46:07.343070 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:46:07.343 UTC [11145] LOG: received smart shutdown request
2026-09-28 22:46:07.346 UTC [11145] LOG: background worker "logical replication launcher" (PID 11154) exited with exit code 1
2026-09-28 22:46:07.346 UTC [11149] LOG: shutting down
2026-09-28 22:46:07.496 UTC [11145] LOG: database system is shut down
2026-09-28T22:46:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44610 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:46:07 dist-test-slave-pxr1 krb5kdc[12131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:46:07 dist-test-slave-pxr1 krb5kdc[12131](info): set up 2 sockets
Sep 28 22:46:07 dist-test-slave-pxr1 krb5kdc[12131](info): commencing operation
krb5kdc: starting...
W20260928 22:46:08.540408 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:46:08 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:46:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 22:46:08.691501 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskntG_DW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/postgres -l logfile start
2026-09-28 22:46:11.141 UTC [12161] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:46:11.141 UTC [12161] LOG: listening on IPv4 address "127.0.98.20", port 34775
2026-09-28 22:46:11.145 UTC [12161] LOG: listening on Unix socket "/tmp/.s.PGSQL.34775"
2026-09-28 22:46:11.173 UTC [12164] LOG: database system was shut down at 2026-09-28 22:46:09 UTC
2026-09-28 22:46:11.177 UTC [12161] LOG: database system is ready to accept connections
I20260928 22:46:12.040308 392 mini_postgres.cc:96] Postgres bound to 34775
2026-09-28 22:46:12.045 UTC [12172] FATAL: database "slave" does not exist
127.0.98.20:34775 - accepting connections
I20260928 22:46:12.046382 392 mini_ranger.cc:161] Starting Ranger...
I20260928 22:46:12.059979 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:46:12.543020 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:46:12.543092 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin
2026-09-28 22:46:12,806 [I] DB FLAVOR :POSTGRES
2026-09-28 22:46:12,806 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:46:12,807 [I] Checking connection..
2026-09-28 22:46:12,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:46:13,068 [I] Checking connection passed.
2026-09-28 22:46:13,068 [I] --------- Verifying version history table ---------
2026-09-28 22:46:13,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:46:13,326 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:46:13,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:46:13,574 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:46:13,574 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:46:13,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:46:13.820 UTC [12236] WARNING: there is no transaction in progress
2026-09-28 22:46:13,825 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:46:13,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:46:14,082 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:46:14,082 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:46:14,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:46:14,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:46:14,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pxr1.c.gcp-upstream.internal','N') ;"
2026-09-28 22:46:14,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:46:15,030 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:46:15,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:46:15,276 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:46:15,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:46:15,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:46:15,777 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:46:15,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:46:16.781 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.800 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.817 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.835 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.855 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.876 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.945 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.952 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.966 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.976 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:16.987 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17.000 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17.009 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17.015 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17.021 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17.027 UTC [12344] WARNING: there is no transaction in progress
2026-09-28 22:46:17,682 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:46:17,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:46:17,907 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:46:17,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:46:18,175 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:46:18,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:46:18,414 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:46:18,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:46:18,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:46:18,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pxr1.c.gcp-upstream.internal';"
2026-09-28 22:46:19,183 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:46:19,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:46:19,411 [I] DB_PATCHES have already been applied
I20260928 22:46:19.417174 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskntG_DW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:46:19.417243 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 22:46:19.420113 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60019
Sep 28, 2026 10:46:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:46:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:46:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:46:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:46:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:46:20.413846 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.367s user 0.000s sys 0.008s
Sep 28, 2026 10:46:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:46:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 10:46:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:46:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:46:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:46:41.109404 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:46:41 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635601, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:46:41.200115 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:46:41.208959 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:46:41.995182 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:46:41.995266 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms
I20260928 22:46:41.995285 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:34775
2026-09-28 22:46:42,153 [I] DB FLAVOR :POSTGRES
2026-09-28 22:46:42,154 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:46:42,154 [I] Checking connection
2026-09-28 22:46:42,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:46:42,388 [I] connection success
2026-09-28 22:46:42,388 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:46:42,388 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:46:42,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:46:42,641 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:46:42,642 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:46:42,642 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:46:42,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:46:42,919 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:46:43.385668 392 mini_ranger_kms.cc:310] Created kms service
I20260928 22:46:43.619210 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:46:43.662019 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:46:43 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635601, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:44.069005 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:46:44.069221 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskntG_DW/build/release/bin/postgresql.jar:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskntG_DW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:46:44.069247 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 22:46:44.072417 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12524
I20260928 22:46:44.072491 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44977
Sep 28, 2026 10:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:46:46.059515 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.859s user 0.000s sys 0.143s
I20260928 22:46:46.059593 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:46:46.059636 392 mini_ranger_kms.cc:401] 127.0.98.20:44977/kms/v1/keys
Sep 28 22:46:46 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635606, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:46:46 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635606, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 22:46:47 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635606, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 22:46:49 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:46:49.235598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:49.316269 12575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:49.316499 12575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:49.316521 12575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:49.317692 12575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:49.317734 12575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:49.317746 12575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:49.317776 12575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:49.317787 12575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:49.319299 12575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44977/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:49.319500 12575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:49.319675 12575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:49.324571 12581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:49.324575 12575 server_base.cc:1034] running on GCE node
W20260928 22:46:49.324672 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:49.324672 12580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:49.325016 12575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:49.325295 12575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:49.326432 12575 hybrid_clock.cc:648] HybridClock initialized: now 1790635609326416 us; error 31 us; skew 500 ppm
Sep 28 22:46:49 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:49.332360 12575 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:49.332847 12575 webserver.cc:460] Webserver started at http://127.0.98.62:42493/ using document root <none> and password file <none>
I20260928 22:46:49.333055 12575 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:49.333094 12575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:49.333217 12575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:49 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:49.387286 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "7354074aa6924edfa520a45fb0908877"
format_stamp: "Formatted at 2026-09-28 22:46:49 on dist-test-slave-pxr1"
server_key: "5be4b2f03985f0250428575928c18416"
server_key_iv: "1a299359431dc7252576cb038ef23e43"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:49.387763 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "7354074aa6924edfa520a45fb0908877"
format_stamp: "Formatted at 2026-09-28 22:46:49 on dist-test-slave-pxr1"
server_key: "5be4b2f03985f0250428575928c18416"
server_key_iv: "1a299359431dc7252576cb038ef23e43"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:49.389503 12575 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:49.403867 12590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:49.404089 12575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:49.404178 12575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7354074aa6924edfa520a45fb0908877"
format_stamp: "Formatted at 2026-09-28 22:46:49 on dist-test-slave-pxr1"
server_key: "5be4b2f03985f0250428575928c18416"
server_key_iv: "1a299359431dc7252576cb038ef23e43"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:49.404266 12575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:49.456701 12575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:49.458824 12575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:49.458982 12575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:49.464108 12575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36111
I20260928 22:46:49.464135 12642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36111 every 8 connection(s)
I20260928 22:46:49.464583 12575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:49.465292 12643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:49.467525 12643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Bootstrap starting.
I20260928 22:46:49.468088 12643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:49.468351 12643 log.cc:824] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:49.468989 12643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: No bootstrap required, opened a new log
I20260928 22:46:49.471004 12643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:49.471154 12643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:49.471176 12643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7354074aa6924edfa520a45fb0908877, State: Initialized, Role: FOLLOWER
I20260928 22:46:49.471310 12643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:49.471377 12643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:49.471410 12643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:49.471464 12643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:49.471978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 12575
I20260928 22:46:49.472057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:46:49.472024 12643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:49.472121 12643 leader_election.cc:302] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7354074aa6924edfa520a45fb0908877; no voters:
I20260928 22:46:49.472209 392 external_mini_cluster.cc:1428] Setting key 71ce98da13afda0f2e027d7302ebae3c
I20260928 22:46:49.472276 12643 leader_election.cc:288] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:46:49.472332 12646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:49.472491 12646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 LEADER]: Becoming Leader. State: Replica: 7354074aa6924edfa520a45fb0908877, State: Running, Role: LEADER
I20260928 22:46:49.472533 12643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:49.472633 12646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:49.473170 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7354074aa6924edfa520a45fb0908877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260928 22:46:49.473380 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:49.473320 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7354074aa6924edfa520a45fb0908877. Latest consensus state: current_term: 1 leader_uuid: "7354074aa6924edfa520a45fb0908877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260928 22:46:49.473651 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:49.473809 12659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:49.473973 12643 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:46:49.474344 12659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:46:49.478907 12659 catalog_manager.cc:1347] Generated new cluster ID: 64bd1d5ebd604873970c487eea6de340
I20260928 22:46:49.478984 12659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:46:49 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:49.486438 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:49.472791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.62:36111)
0928 22:46:49.472986 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:49.472996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:49.473030 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:46:49.473148 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:49.473156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:49.473215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:49.473952 (+ 737us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:49.474161 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:49.477412 (+ 3251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:49.478582 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:49.479261 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:49.479706 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:49.482204 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:49.482210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:49.482210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:49.482225 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:49.484244 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:49.484680 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:49.484683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:49.484683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:49.484824 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:49.485045 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:49.485046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:49.485046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:49.485371 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:49.485980 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:49.486053 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:46:49.486188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 22:46:49.493759 12659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:46:49.494467 12659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:49.501641 12659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Generated new TSK 0
I20260928 22:46:49.501843 12659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:46:50.184253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:50.297219 12688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:50.297575 12688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:50.297624 12688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:50.299567 12688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:50.299645 12688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:50.299690 12688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:46:50.302096 12688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:50.302402 12688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:50.302700 12688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:50.309152 12696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:50.309170 12693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:50.310339 12688 server_base.cc:1034] running on GCE node
W20260928 22:46:50.310774 12694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:50.311061 12688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:50.311342 12688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:50.312523 12688 hybrid_clock.cc:648] HybridClock initialized: now 1790635610312505 us; error 37 us; skew 500 ppm
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:50.315975 12688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:46:50.316617 12688 webserver.cc:460] Webserver started at http://127.0.98.1:44803/ using document root <none> and password file <none>
I20260928 22:46:50.316911 12688 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:50.316987 12688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:50.317137 12688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:50.337426 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "98144030a2794edba42ad9df762ed328"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "7b4229efee7e848f7b3ae2ac70d67cf5"
server_key_iv: "ef43a85d749ab397937097044734c957"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.338016 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98144030a2794edba42ad9df762ed328"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "7b4229efee7e848f7b3ae2ac70d67cf5"
server_key_iv: "ef43a85d749ab397937097044734c957"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.339843 12688 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:50.356072 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.356328 12688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 22:46:50.356433 12688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "98144030a2794edba42ad9df762ed328"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "7b4229efee7e848f7b3ae2ac70d67cf5"
server_key_iv: "ef43a85d749ab397937097044734c957"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.356542 12688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:50.369130 12688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:50.371022 12688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:50.371469 12688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:50.371701 12688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:50.372057 12688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:50.372100 12688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.372153 12688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:50.372193 12688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.378856 12688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34307
I20260928 22:46:50.379383 12688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:46:50.386515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 12688
I20260928 22:46:50.386632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:46:50.386821 392 external_mini_cluster.cc:1428] Setting key 516803c5c454aea55110c8865afc56df
I20260928 22:46:50.394351 12818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34307 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 22:46:50.417663 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.394801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.1:60409)
0928 22:46:50.395040 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:50.395094 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:50.395748 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:50.396159 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:50.396165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:50.396427 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:50.396758 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.396764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.397809 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.397810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.399048 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.399051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.399563 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.399564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.399703 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.400296 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:50.400306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:50.407750 (+ 7444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:50.415839 (+ 8089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.415853 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.415856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.416181 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.416438 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.416439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.416439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.416593 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.416918 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:50.416925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:50.417117 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 22:46:50.417241 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:46:50.417416 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":131,"threads_started":1}
I20260928 22:46:50.418469 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.395166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60409 (local address 127.0.98.62:36111)
0928 22:46:50.395942 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:50.395944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:50.395952 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:50.395977 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:50.395978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:50.395991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:50.396853 (+ 862us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:50.396894 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.397678 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.399137 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.399453 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.399945 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.408793 (+ 8848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:50.408795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:50.408795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:50.408804 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:50.415740 (+ 6936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.416290 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.416291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.416292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.416362 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.416669 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.416670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.416670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.416841 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:50.418233 (+ 1392us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:50.418277 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:46:50.418366 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":77,"threads_started":1}
I20260928 22:46:50.419027 12819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:50.419127 12819 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:50.419301 12819 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:50.419822 12607 ts_manager.cc:194] Registered new tserver with Master: 98144030a2794edba42ad9df762ed328 (127.0.98.1:34307)
I20260928 22:46:50.420758 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:46:50.431134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 22:46:50.544076 12827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:50.544415 12827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:50.544479 12827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:50.546352 12827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:50.546458 12827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:50.546530 12827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:46:50.548890 12827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:50.549261 12827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:50.549552 12827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:50.556094 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:50.556509 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:50.556702 12827 server_base.cc:1034] running on GCE node
W20260928 22:46:50.556790 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:50.557181 12827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:50.557459 12827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:50.560745 12827 hybrid_clock.cc:648] HybridClock initialized: now 1790635610560686 us; error 43 us; skew 500 ppm
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:50.563992 12827 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:46:50.564534 12827 webserver.cc:460] Webserver started at http://127.0.98.2:44941/ using document root <none> and password file <none>
I20260928 22:46:50.564828 12827 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:50.564893 12827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:50.565022 12827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:50.581472 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1c3ee38dd244a73b8beef4eda361dcc"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "79e5dec8a08ca867ce47c068a15c2d0d"
server_key_iv: "3311e12f3b7a0f1eafc22c875e50fc45"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.581924 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1c3ee38dd244a73b8beef4eda361dcc"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "79e5dec8a08ca867ce47c068a15c2d0d"
server_key_iv: "3311e12f3b7a0f1eafc22c875e50fc45"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.587210 12827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 22:46:50.605317 12843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.605581 12827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:50.605665 12827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "b1c3ee38dd244a73b8beef4eda361dcc"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "79e5dec8a08ca867ce47c068a15c2d0d"
server_key_iv: "3311e12f3b7a0f1eafc22c875e50fc45"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.605757 12827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:50.617293 12827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:50.619643 12827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:50.619832 12827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:50.620065 12827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:50.620462 12827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:50.620545 12827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.620611 12827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:50.620666 12827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.627240 12827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38875
I20260928 22:46:50.627741 12827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:46:50.630286 12956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38875 every 8 connection(s)
I20260928 22:46:50.637632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 12827
I20260928 22:46:50.637756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:46:50.637941 392 external_mini_cluster.cc:1428] Setting key 53cff4e28aa6824de46dea428b760727
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:50.642064 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.631172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50045 (local address 127.0.98.62:36111)
0928 22:46:50.631237 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:50.631239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:50.631734 (+ 495us) server_negotiation.cc:407] Connection header received
0928 22:46:50.632427 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:50.632428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:50.632440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:50.632469 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:50.633314 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.634120 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.635663 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.636000 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.636046 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.638736 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:50.638737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:50.638737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:50.638745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:50.640364 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.640958 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.640960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.640960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.641021 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.641305 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.641306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.641306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.641458 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:50.641507 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:50.641875 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 22:46:50.641971 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:46:50.642843 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.630886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.2:50045)
0928 22:46:50.631602 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:50.631645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:50.632293 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:50.632652 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:50.632656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:50.632925 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:50.633202 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.633208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.634235 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.634236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.635560 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.635563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.636148 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.636149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.636297 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.636933 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:50.636943 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:50.638639 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:50.640515 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.640522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.640525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.640868 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.641092 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.641093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.641093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.641229 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.641540 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:50.641546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:50.641726 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 22:46:50.642438 (+ 712us) client_negotiation.cc:240] Negotiation successful
0928 22:46:50.642612 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":90,"threads_started":1}
I20260928 22:46:50.643467 12957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:50.643565 12957 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:50.643743 12957 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:50.644140 12607 ts_manager.cc:194] Registered new tserver with Master: b1c3ee38dd244a73b8beef4eda361dcc (127.0.98.2:38875)
I20260928 22:46:50.644639 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50045
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:46:50.689522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:50.801608 12962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:50.801887 12962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:50.801925 12962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:50.803690 12962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:50.803791 12962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:50.803851 12962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:46:50.806177 12962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:50.806460 12962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:50.806772 12962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:50.813339 12970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:50.813522 12967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:50.813337 12968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:50.813807 12962 server_base.cc:1034] running on GCE node
I20260928 22:46:50.814026 12962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:50.814320 12962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:50.817600 12962 hybrid_clock.cc:648] HybridClock initialized: now 1790635610817568 us; error 48 us; skew 500 ppm
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:50.821048 12962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:46:50.821609 12962 webserver.cc:460] Webserver started at http://127.0.98.3:46867/ using document root <none> and password file <none>
I20260928 22:46:50.821913 12962 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:50.822017 12962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:50.822182 12962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:50.837787 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "af66e788f5b743b1b8832962a16c8ef7"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "bea79af977929359f223576eec09e49c"
server_key_iv: "c86b942ca81f106078fd2e0b3f80411a"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.838374 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af66e788f5b743b1b8832962a16c8ef7"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "bea79af977929359f223576eec09e49c"
server_key_iv: "c86b942ca81f106078fd2e0b3f80411a"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.843240 12962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 22:46:50.857709 12977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.857964 12962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:50.858045 12962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "af66e788f5b743b1b8832962a16c8ef7"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "bea79af977929359f223576eec09e49c"
server_key_iv: "c86b942ca81f106078fd2e0b3f80411a"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:50.858132 12962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:50.881011 12962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:50.883368 12962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:50.883518 12962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:50.883718 12962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:50.884075 12962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:50.884119 12962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.884148 12962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:50.884171 12962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:50.890602 12962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37303
I20260928 22:46:50.891110 12962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:46:50.892473 13090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37303 every 8 connection(s)
I20260928 22:46:50.901306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 12962
I20260928 22:46:50.901425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:46:50.901585 392 external_mini_cluster.cc:1428] Setting key 948db0d35db8b973d8097d44c623ceb6
Sep 28 22:46:50 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:50.909761 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.891924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.3:48629)
0928 22:46:50.896857 (+ 4933us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:50.896925 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:50.897590 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:50.897910 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:50.897915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:50.898170 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:50.898456 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.898462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.899393 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.899394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.900679 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:50.900683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.904169 (+ 3486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.904171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:50.904357 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.905015 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:50.905027 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:50.906448 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:50.908084 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.908093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.908095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.908375 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.908570 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:50.908571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:50.908571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:50.908695 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:50.908988 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:50.908996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:50.909194 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 22:46:50.909300 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:46:50.909477 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4863,"thread_start_us":88,"threads_started":1}
I20260928 22:46:50.909780 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:50.892232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48629 (local address 127.0.98.62:36111)
0928 22:46:50.892279 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:50.892281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:50.897057 (+ 4776us) server_negotiation.cc:407] Connection header received
0928 22:46:50.897729 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:50.897731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:50.897746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:50.897823 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:50.898564 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.899297 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.900816 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:50.901127 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:50.901201 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:50.906586 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:50.906588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:50.906589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:50.906598 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:50.908002 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.908449 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.908450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.908451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.908516 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:50.908779 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:50.908780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:50.908781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:50.908929 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:50.908990 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:50.909479 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 22:46:50.909656 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:46:50.910681 13091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:50.910784 13091 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:50.910976 13091 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:50.911332 12607 ts_manager.cc:194] Registered new tserver with Master: af66e788f5b743b1b8832962a16c8ef7 (127.0.98.3:37303)
I20260928 22:46:50.911693 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48629
I20260928 22:46:50.913147 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:46:50.913712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 12688
I20260928 22:46:50.921183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 12827
I20260928 22:46:50.927528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 12962
I20260928 22:46:50.932797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 12575
I20260928 22:46:50.937486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--webserver_interface=127.0.98.62
--webserver_port=42493
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:51.017971 13094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:51.018179 13094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:51.018199 13094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:51.019356 13094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:51.019402 13094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:51.019415 13094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:51.019426 13094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:51.019438 13094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:51.021083 13094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:51.021292 13094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:51.021505 13094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:51.026147 13102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.026170 13099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.026167 13100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:51.026317 13094 server_base.cc:1034] running on GCE node
I20260928 22:46:51.026539 13094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:51.026860 13094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:51.028029 13094 hybrid_clock.cc:648] HybridClock initialized: now 1790635611028014 us; error 28 us; skew 500 ppm
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:51.030398 13094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:51.030864 13094 webserver.cc:460] Webserver started at http://127.0.98.62:42493/ using document root <none> and password file <none>
I20260928 22:46:51.031069 13094 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:51.031122 13094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:51.088897 13094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:51.089790 13110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.089964 13094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:51.090070 13094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7354074aa6924edfa520a45fb0908877"
format_stamp: "Formatted at 2026-09-28 22:46:49 on dist-test-slave-pxr1"
server_key: "5be4b2f03985f0250428575928c18416"
server_key_iv: "1a299359431dc7252576cb038ef23e43"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:51.090536 13094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:51.119997 13094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:51.121613 13094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:51.121726 13094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:51.125486 13094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36111
I20260928 22:46:51.125541 13162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36111 every 8 connection(s)
I20260928 22:46:51.125885 13094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:51.127041 13163 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:46:51.127658 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Bootstrap starting.
I20260928 22:46:51.133917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13094
I20260928 22:46:51.134183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34307
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36111
--webserver_port=44803
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:51.135309 13163 log.cc:824] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:51.137079 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:46:51.137323 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Bootstrap complete.
I20260928 22:46:51.139459 13163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:51.139688 13163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7354074aa6924edfa520a45fb0908877, State: Initialized, Role: FOLLOWER
I20260928 22:46:51.139833 13163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:51.139906 13163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:51.139950 13163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:51.139997 13163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:46:51.141040 13163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:51.141137 13163 leader_election.cc:302] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7354074aa6924edfa520a45fb0908877; no voters:
I20260928 22:46:51.141307 13163 leader_election.cc:288] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:46:51.141359 13167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:46:51.141464 13167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [term 2 LEADER]: Becoming Leader. State: Replica: 7354074aa6924edfa520a45fb0908877, State: Running, Role: LEADER
I20260928 22:46:51.141564 13163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:51.141549 13167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260928 22:46:51.141964 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7354074aa6924edfa520a45fb0908877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260928 22:46:51.142064 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:51.141980 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7354074aa6924edfa520a45fb0908877. Latest consensus state: current_term: 2 leader_uuid: "7354074aa6924edfa520a45fb0908877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7354074aa6924edfa520a45fb0908877" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260928 22:46:51.142230 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:51.142359 13175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:51.142877 13175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:46:51.143005 13175 catalog_manager.cc:1259] Loaded cluster ID: 64bd1d5ebd604873970c487eea6de340
I20260928 22:46:51.143036 13175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:51.143249 13163 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:46:51.146014 13175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:51.146584 13175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Loaded TSK: 0
I20260928 22:46:51.146867 13175 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:46:51.238695 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:51.238905 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:51.238938 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:51.240129 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:51.240172 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:51.240208 13165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:46:51.241835 13165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44803
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:51.242048 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:51.242286 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:51.248587 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.248587 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.248610 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:51.249657 13165 server_base.cc:1034] running on GCE node
I20260928 22:46:51.249874 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:51.250165 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:51.251343 13165 hybrid_clock.cc:648] HybridClock initialized: now 1790635611251316 us; error 37 us; skew 500 ppm
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:51.254513 13165 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:46:51.255070 13165 webserver.cc:460] Webserver started at http://127.0.98.1:44803/ using document root <none> and password file <none>
I20260928 22:46:51.255338 13165 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:51.255415 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:51.277895 13165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 22:46:51.278656 13211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.278797 13165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:46:51.278877 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "98144030a2794edba42ad9df762ed328"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "7b4229efee7e848f7b3ae2ac70d67cf5"
server_key_iv: "ef43a85d749ab397937097044734c957"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:51.279351 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:51.302839 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:51.305198 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:51.305362 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:51.305583 13165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:51.305958 13165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:51.306006 13165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.306036 13165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:51.306082 13165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.312203 13165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34307
I20260928 22:46:51.312279 13324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34307 every 8 connection(s)
I20260928 22:46:51.312669 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:51.321411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13165
I20260928 22:46:51.321707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38875
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36111
--webserver_port=44941
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:51.324246 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.313438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.1:50059)
0928 22:46:51.313722 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:51.313756 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:51.314216 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:51.314720 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:51.314723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:51.314900 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:51.315085 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.315089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.316780 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.316781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.317596 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.317599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.318265 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.318266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.318414 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.318977 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:51.318988 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:51.320359 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:51.322349 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.322357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.322359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.322676 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.322960 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.322962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.322962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.323106 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.323591 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:51.323597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:51.323760 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 22:46:51.323869 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:46:51.324028 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":80,"threads_started":1}
I20260928 22:46:51.324874 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.313530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50059 (local address 127.0.98.62:36111)
0928 22:46:51.313729 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:51.313739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:51.313842 (+ 103us) server_negotiation.cc:407] Connection header received
0928 22:46:51.314445 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:51.314452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:51.314513 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:51.314681 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:51.315209 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.316621 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.317727 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.318164 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.318271 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.320473 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:51.320477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:51.320477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:51.320488 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:51.322257 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.322763 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.322767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.322768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.322888 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.323173 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.323174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.323174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.323487 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:51.323562 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:51.324537 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 22:46:51.324657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260928 22:46:51.325791 13325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:51.325902 13325 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.326099 13325 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.328280 13325 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:46:51.328359 13325 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.328429 13325 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:51.331789 13325 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.331883 13325 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.332266 13325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:46:51.430159 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:51.430382 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:51.430405 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:51.431646 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:51.431689 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:51.431726 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:46:51.433357 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44941
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:51.433604 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:51.433861 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:51.438966 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.438952 13334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.439658 13335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:51.440084 13329 server_base.cc:1034] running on GCE node
I20260928 22:46:51.440300 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:51.440531 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:51.447858 13329 hybrid_clock.cc:648] HybridClock initialized: now 1790635611447836 us; error 33 us; skew 500 ppm
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:51.450515 13329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:46:51.450978 13329 webserver.cc:460] Webserver started at http://127.0.98.2:44941/ using document root <none> and password file <none>
I20260928 22:46:51.451202 13329 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:51.451257 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:51.477243 13329 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260928 22:46:51.478579 13345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.478790 13329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:46:51.478880 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "b1c3ee38dd244a73b8beef4eda361dcc"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "79e5dec8a08ca867ce47c068a15c2d0d"
server_key_iv: "3311e12f3b7a0f1eafc22c875e50fc45"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:51.479362 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:51.508409 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:51.510591 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:51.510747 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:51.511024 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:51.511369 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:51.511406 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.511436 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:51.511458 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.518044 13329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38875
I20260928 22:46:51.518445 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:46:51.518576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13329
I20260928 22:46:51.518863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37303
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36111
--webserver_port=46867
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:51.518970 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38875 every 8 connection(s)
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:51.529227 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.521270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48625 (local address 127.0.98.62:36111)
0928 22:46:51.521325 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:51.521328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:51.521519 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:46:51.522099 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:51.522100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:51.522116 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:51.522160 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:51.522706 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.523207 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.524160 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.524374 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.524431 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.526536 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:51.526538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:51.526538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:51.526546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:51.527806 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.528175 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.528176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.528177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.528230 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.528447 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.528448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.528448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.528569 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:51.528627 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:51.529036 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 22:46:51.529127 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:46:51.529856 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.521244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.2:48625)
0928 22:46:51.521397 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:51.521439 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:51.521992 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:51.522237 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:51.522241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:51.522424 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:51.522613 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.522617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.523287 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.523288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.524080 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.524082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.524456 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.524457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.524569 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.525049 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:51.525058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:51.526449 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:51.527882 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.527890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.527893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.528110 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.528285 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.528285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.528286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.528394 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.528654 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:51.528660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:51.528875 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 22:46:51.529544 (+ 669us) client_negotiation.cc:240] Negotiation successful
0928 22:46:51.529689 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260928 22:46:51.530416 13459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:51.530512 13459 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.530694 13459 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.531301 13459 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:46:51.531363 13459 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.531427 13459 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:51.531711 13459 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.531785 13459 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.532051 13459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:46:51.637193 13461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:51.637388 13461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:51.637411 13461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:51.638871 13461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:51.638968 13461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:51.639026 13461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:46:51.641275 13461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46867
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:51.641664 13461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:51.641944 13461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:51.648940 13468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:51.648945 13469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:51.649302 13461 server_base.cc:1034] running on GCE node
W20260928 22:46:51.652870 13471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:51.653131 13461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:51.653424 13461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:51.654579 13461 hybrid_clock.cc:648] HybridClock initialized: now 1790635611654572 us; error 29 us; skew 500 ppm
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:51.657011 13461 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:46:51.657469 13461 webserver.cc:460] Webserver started at http://127.0.98.3:46867/ using document root <none> and password file <none>
I20260928 22:46:51.657701 13461 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:51.657763 13461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:51.679373 13461 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 22:46:51.680069 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.680228 13461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:46:51.680327 13461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "af66e788f5b743b1b8832962a16c8ef7"
format_stamp: "Formatted at 2026-09-28 22:46:50 on dist-test-slave-pxr1"
server_key: "bea79af977929359f223576eec09e49c"
server_key_iv: "c86b942ca81f106078fd2e0b3f80411a"
server_key_version: "kuduclusterkey@0"
I20260928 22:46:51.680838 13461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:51.701170 13461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:51.703040 13461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:51.703174 13461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:51.703397 13461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:51.703727 13461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:51.703763 13461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.703785 13461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:51.703799 13461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:51.710214 13461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37303
I20260928 22:46:51.710600 13461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:46:51.710996 13592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37303 every 8 connection(s)
I20260928 22:46:51.714877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13461
Sep 28 22:46:51 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:51.720743 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.711590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55901 (local address 127.0.98.62:36111)
0928 22:46:51.711659 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:51.711661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:51.711671 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:46:51.712252 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:51.712254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:51.712266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:51.712316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:51.712934 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.713685 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.714734 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.715031 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.715106 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.717654 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:51.717656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:51.717656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:51.717664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:51.719298 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.719759 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.719760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.719761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.719817 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.720080 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.720081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.720082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.720201 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:51.720269 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:51.720547 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 22:46:51.720627 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:46:51.720929 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.711312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.3:55901)
0928 22:46:51.711453 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:51.711493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:51.712156 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:51.712386 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:51.712390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:51.712567 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:51.712829 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.712836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.713801 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.713802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.714622 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:51.714624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.715187 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.715189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:51.715338 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.715914 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:51.715925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:51.717524 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:51.719389 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.719397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.719399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.719684 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.719882 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:51.719883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:51.719883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:51.720013 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:51.720269 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:51.720276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:51.720422 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 22:46:51.720514 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:46:51.720662 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260928 22:46:51.721431 13593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260928 22:46:51.721547 13593 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.721762 13593 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.722352 13593 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:46:51.722438 13593 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.722512 13593 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:51.722795 13593 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:51.722879 13593 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
W20260928 22:46:51.723136 13593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:46:51.725458 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:51.716842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.0.98.62:36111)
0928 22:46:51.718044 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:51.718046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:51.718055 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:46:51.718076 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:51.718077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:51.718088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:51.718977 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:51.719015 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.719753 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.720875 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:51.721171 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:51.722621 (+ 1450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:51.722672 (+ 51us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:51.722673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:51.722673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:51.722678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:51.724323 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.724654 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.724655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.724655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.724750 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:51.724929 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:51.724930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:51.724931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:51.725066 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:51.725243 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:51.725281 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:46:51.725362 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":53,"threads_started":1}
Sep 28 22:46:52 dist-test-slave-pxr1 krb5kdc[12131](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:52.332448 13325 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:52.332597 13325 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:52.332970 13127 ts_manager.cc:194] Registered new tserver with Master: 98144030a2794edba42ad9df762ed328 (127.0.98.1:34307)
I20260928 22:46:52.333663 13127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50059
I20260928 22:46:52.532218 13459 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:52.532375 13459 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:52.532678 13127 ts_manager.cc:194] Registered new tserver with Master: b1c3ee38dd244a73b8beef4eda361dcc (127.0.98.2:38875)
I20260928 22:46:52.533250 13127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48625
Sep 28 22:46:52 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:52.723343 13593 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:52.723515 13593 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260928 22:46:52.723902 13127 ts_manager.cc:194] Registered new tserver with Master: af66e788f5b743b1b8832962a16c8ef7 (127.0.98.3:37303)
I20260928 22:46:52.724403 13127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55901
I20260928 22:46:52.729817 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:46:52.739522 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.730768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.62:36111)
0928 22:46:52.731057 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.731059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.731070 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:46:52.731093 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.731094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.731105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.731170 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:52.731864 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.732636 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.733648 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.733979 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.734055 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.736533 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:52.736535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:52.736535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:52.736544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:52.738392 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:52.738725 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:52.738727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:52.738727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:52.738785 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:52.738945 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:52.738946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:52.738946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:52.739058 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:52.739142 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.739262 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.739342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":70,"threads_started":1}
I20260928 22:46:52.741417 13127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:46:52.743317 13127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:46:52.753218 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.746236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37303 (local address 127.0.0.1:58228)
0928 22:46:52.748942 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.748950 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:52.748989 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:52.749253 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:52.749254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:52.749256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:52.749287 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.749288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.750535 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.750536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.751841 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.751843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.752919 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.752920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.753000 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.753014 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:46:52.753075 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:46:52.753124 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2695,"mutex_wait_us":26,"thread_start_us":62,"threads_started":1}
I20260928 22:46:52.753958 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.746317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.98.3:37303)
0928 22:46:52.746423 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.746426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.749048 (+ 2622us) server_negotiation.cc:407] Connection header received
0928 22:46:52.749086 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.749088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.749102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.749163 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:52.749353 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.750445 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.751937 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.752682 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.752776 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.752859 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.753826 (+ 967us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.753878 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":66,"threads_started":1}
I20260928 22:46:52.754724 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.745837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34307 (local address 127.0.0.1:37154)
0928 22:46:52.745976 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.746015 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:52.746080 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:52.751257 (+ 5177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:52.751258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:52.751260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:52.751302 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.751304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.752568 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.752569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.753599 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.753603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.754517 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.754518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.754580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.754585 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:46:52.754612 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:46:52.754649 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 22:46:52.755053 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.745905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.0.98.1:34307)
0928 22:46:52.751072 (+ 5167us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.751078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.751089 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:46:52.751124 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.751126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.751143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.751363 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:52.751445 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.752474 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.753691 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.754421 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.754842 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.754916 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.754942 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.754983 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5124,"thread_start_us":57,"threads_started":1}
I20260928 22:46:52.754958 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f (DEFAULT_TABLE table=test-table [id=cd3e2c7a22e74dafad214944b9f886e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:52.755292 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389af075eed4431c9b6b0b91ca1eec7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:52.758045 13626 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Bootstrap starting.
I20260928 22:46:52.758705 13626 tablet_bootstrap.cc:654] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:52.758759 13259 tablet_service.cc:1467] Processing CreateTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f (DEFAULT_TABLE table=test-table [id=cd3e2c7a22e74dafad214944b9f886e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:52.758982 13626 log.cc:824] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:52.759092 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389af075eed4431c9b6b0b91ca1eec7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:52.763079 13626 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: No bootstrap required, opened a new log
I20260928 22:46:52.763159 13626 ts_tablet_manager.cc:1397] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 22:46:52.763770 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.746062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38875 (local address 127.0.0.1:32932)
0928 22:46:52.748808 (+ 2746us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.748818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:52.748871 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:52.749142 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:52.749146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:52.749148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:52.749224 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.749227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.750538 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.750538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.762401 (+ 11863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.762405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.763486 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.763488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.763563 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.763567 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:46:52.763617 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:46:52.763667 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2727,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260928 22:46:52.764066 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.746151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.98.2:38875)
0928 22:46:52.746340 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.746344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.748885 (+ 2541us) server_negotiation.cc:407] Connection header received
0928 22:46:52.748934 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.748937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.748958 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.749034 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:52.749307 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.750431 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.762504 (+ 12073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.763281 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.763359 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.763439 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.763931 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.763986 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":50,"threads_started":1}
I20260928 22:46:52.764755 13393 tablet_service.cc:1467] Processing CreateTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f (DEFAULT_TABLE table=test-table [id=cd3e2c7a22e74dafad214944b9f886e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:52.765090 13393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389af075eed4431c9b6b0b91ca1eec7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:52.766324 13626 raft_consensus.cc:348] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.766495 13626 raft_consensus.cc:374] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:52.766530 13626 raft_consensus.cc:729] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af66e788f5b743b1b8832962a16c8ef7, State: Initialized, Role: FOLLOWER
I20260928 22:46:52.766651 13626 consensus_queue.cc:260] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.766995 13626 ts_tablet_manager.cc:1428] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:46:52.767176 13593 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260928 22:46:52.767737 13628 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Bootstrap starting.
I20260928 22:46:52.768432 13628 tablet_bootstrap.cc:654] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:52.768777 13628 log.cc:824] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:52.769641 13628 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: No bootstrap required, opened a new log
I20260928 22:46:52.769713 13628 ts_tablet_manager.cc:1397] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:52.769994 13631 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: Bootstrap starting.
I20260928 22:46:52.770677 13631 tablet_bootstrap.cc:654] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:52.771026 13631 log.cc:824] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:52.771783 13631 tablet_bootstrap.cc:492] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: No bootstrap required, opened a new log
I20260928 22:46:52.771844 13631 ts_tablet_manager.cc:1397] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:46:52.772372 13628 raft_consensus.cc:348] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.772547 13628 raft_consensus.cc:374] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:52.772578 13628 raft_consensus.cc:729] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98144030a2794edba42ad9df762ed328, State: Initialized, Role: FOLLOWER
I20260928 22:46:52.772697 13628 consensus_queue.cc:260] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.772985 13628 ts_tablet_manager.cc:1428] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:46:52.773350 13325 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260928 22:46:52.774325 13631 raft_consensus.cc:348] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.774545 13631 raft_consensus.cc:374] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:52.774664 13631 raft_consensus.cc:729] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c3ee38dd244a73b8beef4eda361dcc, State: Initialized, Role: FOLLOWER
I20260928 22:46:52.774930 13631 consensus_queue.cc:260] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.775221 13459 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260928 22:46:52.775429 13631 ts_tablet_manager.cc:1428] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260928 22:46:52.814126 13326 tablet.cc:2369] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:46:52.891361 13633 raft_consensus.cc:482] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:46:52.891517 13633 raft_consensus.cc:504] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.891933 13633 leader_election.cc:288] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1c3ee38dd244a73b8beef4eda361dcc (127.0.98.2:38875), af66e788f5b743b1b8832962a16c8ef7 (127.0.98.3:37303)
Sep 28 22:46:52 dist-test-slave-pxr1 krb5kdc[12131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 22:46:52.896466 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.892669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55113 (local address 127.0.98.2:38875)
0928 22:46:52.892739 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.892741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.892937 (+ 196us) server_negotiation.cc:407] Connection header received
0928 22:46:52.893022 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.893023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.893035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.893061 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:52.893269 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.893977 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.895492 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.896084 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.896133 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.896145 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.896346 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.896391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:46:52.896930 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.892475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38875 (local address 127.0.98.1:55113)
0928 22:46:52.892890 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.892898 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:52.892958 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:52.893119 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:52.893120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:52.893122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:52.893188 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.893190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.894184 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.894185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.895384 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.895388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.896203 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.896204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.896273 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.896276 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:46:52.896778 (+ 502us) client_negotiation.cc:240] Negotiation successful
0928 22:46:52.896826 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":82,"threads_started":1}
I20260928 22:46:52.897409 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389af075eed4431c9b6b0b91ca1eec7f" candidate_uuid: "98144030a2794edba42ad9df762ed328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" is_pre_election: true
I20260928 22:46:52.897605 13410 raft_consensus.cc:2454] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98144030a2794edba42ad9df762ed328 in term 0.
I20260928 22:46:52.897680 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.892368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36077 (local address 127.0.98.3:37303)
0928 22:46:52.892461 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.892463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.892793 (+ 330us) server_negotiation.cc:407] Connection header received
0928 22:46:52.892867 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.892868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.892882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.892940 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:52.893143 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.893923 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.896607 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.897259 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.897306 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.897316 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.897563 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.897612 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 22:46:52.897995 13215 leader_election.cc:302] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98144030a2794edba42ad9df762ed328, b1c3ee38dd244a73b8beef4eda361dcc; no voters:
I20260928 22:46:52.898115 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.892143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37303 (local address 127.0.98.1:36077)
0928 22:46:52.892699 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.892709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:52.892807 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:52.892960 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:52.892961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:52.892964 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:52.893059 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.893062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.894034 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.894035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.896504 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:52.896507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.897391 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.897392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:52.897462 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.897465 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:46:52.897974 (+ 509us) client_negotiation.cc:240] Negotiation successful
0928 22:46:52.898021 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":784,"thread_start_us":106,"threads_started":1}
I20260928 22:46:52.898195 13633 raft_consensus.cc:2781] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:46:52.898237 13633 raft_consensus.cc:482] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:46:52.898257 13633 raft_consensus.cc:3037] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:52.898620 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389af075eed4431c9b6b0b91ca1eec7f" candidate_uuid: "98144030a2794edba42ad9df762ed328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af66e788f5b743b1b8832962a16c8ef7" is_pre_election: true
I20260928 22:46:52.898772 13547 raft_consensus.cc:2454] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98144030a2794edba42ad9df762ed328 in term 0.
I20260928 22:46:52.899108 13633 raft_consensus.cc:504] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.899240 13633 leader_election.cc:288] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [CANDIDATE]: Term 1 election: Requested vote from peers b1c3ee38dd244a73b8beef4eda361dcc (127.0.98.2:38875), af66e788f5b743b1b8832962a16c8ef7 (127.0.98.3:37303)
I20260928 22:46:52.900496 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389af075eed4431c9b6b0b91ca1eec7f" candidate_uuid: "98144030a2794edba42ad9df762ed328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af66e788f5b743b1b8832962a16c8ef7"
I20260928 22:46:52.900583 13547 raft_consensus.cc:3037] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:52.901458 13547 raft_consensus.cc:2454] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98144030a2794edba42ad9df762ed328 in term 1.
I20260928 22:46:52.901707 13212 leader_election.cc:302] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98144030a2794edba42ad9df762ed328, af66e788f5b743b1b8832962a16c8ef7; no voters:
I20260928 22:46:52.901693 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389af075eed4431c9b6b0b91ca1eec7f" candidate_uuid: "98144030a2794edba42ad9df762ed328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c3ee38dd244a73b8beef4eda361dcc"
I20260928 22:46:52.901769 13410 raft_consensus.cc:3037] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:52.902558 13410 raft_consensus.cc:2454] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98144030a2794edba42ad9df762ed328 in term 1.
I20260928 22:46:52.902659 13633 raft_consensus.cc:2781] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:52.902804 13633 raft_consensus.cc:686] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 1 LEADER]: Becoming Leader. State: Replica: 98144030a2794edba42ad9df762ed328, State: Running, Role: LEADER
I20260928 22:46:52.902894 13633 consensus_queue.cc:237] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } }
I20260928 22:46:52.903697 13127 catalog_manager.cc:5495] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98144030a2794edba42ad9df762ed328 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "98144030a2794edba42ad9df762ed328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98144030a2794edba42ad9df762ed328" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 } health_report { overall_health: UNKNOWN } } }
I20260928 22:46:52.936414 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.934081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.98.1:34307)
0928 22:46:52.934134 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:52.934136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:52.934143 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:46:52.934182 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:52.934183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:52.934204 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:52.934268 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:46:52.934404 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.934886 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.935630 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:52.935836 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:52.935890 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:52.936064 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:46:52.936188 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:46:52.936270 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:52.936314 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:46:52.936336 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:46:52.940126 13410 raft_consensus.cc:1261] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 1 FOLLOWER]: Refusing update from remote peer 98144030a2794edba42ad9df762ed328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:52.940126 13547 raft_consensus.cc:1261] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 1 FOLLOWER]: Refusing update from remote peer 98144030a2794edba42ad9df762ed328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:52.940479 13633 consensus_queue.cc:1035] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af66e788f5b743b1b8832962a16c8ef7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:52.940894 13633 consensus_queue.cc:1035] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1c3ee38dd244a73b8beef4eda361dcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:52.944576 13643 mvcc.cc:205] Tried to move back new op lower bound from 7334443470599180288 to 7334443470450884608. Current Snapshot: MvccSnapshot[applied={T|T < 7334443470599180288}]
I20260928 22:46:52.945035 13644 mvcc.cc:205] Tried to move back new op lower bound from 7334443470599180288 to 7334443470450884608. Current Snapshot: MvccSnapshot[applied={T|T < 7334443470599180288}]
I20260928 22:46:52.945314 13642 mvcc.cc:205] Tried to move back new op lower bound from 7334443470599180288 to 7334443470450884608. Current Snapshot: MvccSnapshot[applied={T|T < 7334443470599180288}]
I20260928 22:46:52.947466 13126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:52.947561 13126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:52.948427 13126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7354074aa6924edfa520a45fb0908877: Sending DeleteTablet for 3 replicas of tablet 389af075eed4431c9b6b0b91ca1eec7f
I20260928 22:46:52.948967 13393 tablet_service.cc:1514] Processing DeleteTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32932
I20260928 22:46:52.949182 13651 tablet_replica.cc:330] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc: stopping tablet replica
I20260928 22:46:52.949286 13651 raft_consensus.cc:2229] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:46:52.949366 13651 raft_consensus.cc:2258] T 389af075eed4431c9b6b0b91ca1eec7f P b1c3ee38dd244a73b8beef4eda361dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:52.949465 13259 tablet_service.cc:1514] Processing DeleteTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37154
I20260928 22:46:52.949667 13652 tablet_replica.cc:330] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328: stopping tablet replica
I20260928 22:46:52.949769 13652 raft_consensus.cc:2229] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:46:52.949800 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 389af075eed4431c9b6b0b91ca1eec7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58228
I20260928 22:46:52.949924 13652 raft_consensus.cc:2258] T 389af075eed4431c9b6b0b91ca1eec7f P 98144030a2794edba42ad9df762ed328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:52.950376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13165
I20260928 22:46:52.953161 13653 tablet_replica.cc:330] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: stopping tablet replica
I20260928 22:46:52.953289 13653 raft_consensus.cc:2229] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:46:52.953377 13653 raft_consensus.cc:2258] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:52.954193 13653 ts_tablet_manager.cc:1905] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:46:52.955818 13653 ts_tablet_manager.cc:1918] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:46:52.955896 13653 log.cc:1196] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/389af075eed4431c9b6b0b91ca1eec7f
I20260928 22:46:52.956139 13653 ts_tablet_manager.cc:1939] T 389af075eed4431c9b6b0b91ca1eec7f P af66e788f5b743b1b8832962a16c8ef7: Deleting consensus metadata
I20260928 22:46:52.956538 13111 catalog_manager.cc:4841] TS af66e788f5b743b1b8832962a16c8ef7 (127.0.98.3:37303): tablet 389af075eed4431c9b6b0b91ca1eec7f (table test-table [id=cd3e2c7a22e74dafad214944b9f886e7]) successfully deleted
W20260928 22:46:52.960093 13114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34307 (error 108)
I20260928 22:46:52.960888 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.960776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34307 (local address 127.0.0.1:37164)
0928 22:46:52.960814 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.960851 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 22:46:52.960899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13329
W20260928 22:46:52.961788 13114 catalog_manager.cc:4584] TS 98144030a2794edba42ad9df762ed328 (127.0.98.1:34307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 389af075eed4431c9b6b0b91ca1eec7f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34307: connect: Connection refused (error 111)
I20260928 22:46:52.967273 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:52.967155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38875 (local address 127.0.0.1:32944)
0928 22:46:52.967207 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:52.967230 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:46:52.967626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13461
I20260928 22:46:52.973901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13094
I20260928 22:46:52.982393 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:46:53.325995 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:46:53.326097 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:46:53.660019 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:46:53.660 UTC [12161] LOG: received smart shutdown request
2026-09-28 22:46:53.663 UTC [12161] LOG: background worker "logical replication launcher" (PID 12170) exited with exit code 1
2026-09-28 22:46:53.663 UTC [12165] LOG: shutting down
2026-09-28 22:46:53.810 UTC [12161] LOG: database system is shut down
2026-09-28T22:46:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46306 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:46:53 dist-test-slave-pxr1 krb5kdc[13663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:46:53 dist-test-slave-pxr1 krb5kdc[13663](info): set up 2 sockets
Sep 28 22:46:53 dist-test-slave-pxr1 krb5kdc[13663](info): commencing operation
krb5kdc: starting...
W20260928 22:46:54.836347 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:46:54 dist-test-slave-pxr1 krb5kdc[13663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:46:54.947252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:55.027179 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:55.027395 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:55.027417 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:55.028590 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:55.028645 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:55.028658 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:55.028669 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:55.028680 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:55.030277 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:55.030521 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:55.030745 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:55.035470 13681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:55.035495 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:55.035488 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:55.035606 13676 server_base.cc:1034] running on GCE node
I20260928 22:46:55.035969 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:55.036275 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:55.037453 13676 hybrid_clock.cc:648] HybridClock initialized: now 1790635615037436 us; error 33 us; skew 500 ppm
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:55.043155 13676 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:55.043623 13676 webserver.cc:460] Webserver started at http://127.0.98.62:33969/ using document root <none> and password file <none>
I20260928 22:46:55.043824 13676 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:55.043872 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:55.044008 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:55.045127 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "52d9fb89094749908fd4fa60a848099b"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "e4770f986252d2b9cb0743b6c112b432"
server_key_iv: "ba3a84eaf9546361b5cf0513b2d0caad"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.045459 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "52d9fb89094749908fd4fa60a848099b"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "e4770f986252d2b9cb0743b6c112b432"
server_key_iv: "ba3a84eaf9546361b5cf0513b2d0caad"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.046753 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:46:55.047600 13691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:55.047809 13676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:55.047914 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "52d9fb89094749908fd4fa60a848099b"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "e4770f986252d2b9cb0743b6c112b432"
server_key_iv: "ba3a84eaf9546361b5cf0513b2d0caad"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.047987 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:55.061167 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:55.063516 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:55.063647 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:55.067447 13676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44029
I20260928 22:46:55.067524 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44029 every 8 connection(s)
I20260928 22:46:55.067828 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:55.068753 13744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:55.070343 13744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:46:55.071669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13676
I20260928 22:46:55.071770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:46:55.071934 392 external_mini_cluster.cc:1428] Setting key ce5d25b24878f893e12d699ceb389e18
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 22:46:55.081043 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.071048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44029 (local address 127.0.0.1:56906)
0928 22:46:55.071248 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.071287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.071755 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.072078 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.072082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.072409 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.072646 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.072652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.074026 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.074027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.075067 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.075070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.075656 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.075657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.075743 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.076267 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.076278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.077942 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.079718 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.079725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.079727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.079917 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.080114 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.080114 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.080115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.080187 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.080507 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.080513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.080580 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.080667 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.080891 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":58,"thread_start_us":85,"threads_started":1}
I20260928 22:46:55.081043 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.071069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:44029)
0928 22:46:55.071235 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.071242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.071362 (+ 120us) server_negotiation.cc:407] Connection header received
0928 22:46:55.071922 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.071924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.071949 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.072028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.072790 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.073907 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.075175 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.075561 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.075653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.078052 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.078060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.078060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.078066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.079639 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.079985 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.079987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.079988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.080065 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.080230 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.080230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.080230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.080438 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.080515 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.080694 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.080830 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 22:46:55.081838 13744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 22:46:55.082195 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.082085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46524)
0928 22:46:55.082121 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.082168 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 22:46:55.082300 13693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
I20260928 22:46:55.082751 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.082653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46532)
0928 22:46:55.082702 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.082728 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:46:55.082890 13744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37245: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 22:46:55.096320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37245
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:55.132150 13744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } attempt: 1
I20260928 22:46:55.132741 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.132561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46544)
0928 22:46:55.132627 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.132677 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:46:55.133055 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.132972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46550)
0928 22:46:55.133010 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.133036 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:46:55.133153 13744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37245: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
W20260928 22:46:55.176867 13751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:55.177065 13751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:55.177086 13751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:55.178251 13751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:55.178296 13751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:55.178309 13751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:55.178320 13751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:55.178331 13751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:55.179834 13751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:55.180063 13751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:55.180263 13751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:55.184813 13757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:55.184894 13751 server_base.cc:1034] running on GCE node
W20260928 22:46:55.184846 13756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:55.184846 13759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:55.185254 13751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:55.185508 13751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:55.186655 13751 hybrid_clock.cc:648] HybridClock initialized: now 1790635615186649 us; error 31 us; skew 500 ppm
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:55.188979 13751 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 22:46:55.189395 13751 webserver.cc:460] Webserver started at http://127.0.98.61:42863/ using document root <none> and password file <none>
I20260928 22:46:55.189594 13751 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:55.189643 13751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:55.189764 13751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:55.190845 13751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data/instance:
uuid: "ea54ed0a2ca14264932bba6ee8e2add9"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "383405bdcc0c30f6d716c330919d9e24"
server_key_iv: "9d6351e2836f080a85dd33f74da44ee8"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.191180 13751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal/instance:
uuid: "ea54ed0a2ca14264932bba6ee8e2add9"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "383405bdcc0c30f6d716c330919d9e24"
server_key_iv: "9d6351e2836f080a85dd33f74da44ee8"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.192404 13751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:46:55.193111 13766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:55.193251 13751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:55.193332 13751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "ea54ed0a2ca14264932bba6ee8e2add9"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "383405bdcc0c30f6d716c330919d9e24"
server_key_iv: "9d6351e2836f080a85dd33f74da44ee8"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.193390 13751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:55.201308 13744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } attempt: 2
I20260928 22:46:55.201711 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.201600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46556)
0928 22:46:55.201650 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.201683 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:46:55.201999 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.201922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46572)
0928 22:46:55.201954 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.201977 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 22:46:55.202071 13744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37245: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37245: connect: Connection refused (error 111)
I20260928 22:46:55.205899 13751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:55.207605 13751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:55.207739 13751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:55.211402 13751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37245
I20260928 22:46:55.211436 13818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37245 every 8 connection(s)
I20260928 22:46:55.211766 13751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
I20260928 22:46:55.212579 13819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:55.214130 13819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:55.221004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13751
I20260928 22:46:55.221117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-1/wal/instance
I20260928 22:46:55.221312 392 external_mini_cluster.cc:1428] Setting key 121e2f97e6261adcfd3ce91abbb7b40e
I20260928 22:46:55.222563 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.214811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.0.98.62:44029)
0928 22:46:55.214872 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.214874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.215055 (+ 181us) server_negotiation.cc:407] Connection header received
0928 22:46:55.215518 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.215519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.215532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.215583 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.216170 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.216658 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.217657 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.217862 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.217919 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.219962 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.219964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.219964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.219973 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.221277 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.221690 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.221691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.221692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.221757 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.221956 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.221957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.221957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.222086 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.222147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.222357 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.222448 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:46:55.222720 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.214773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44029 (local address 127.0.0.1:56914)
0928 22:46:55.214934 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.214968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.215422 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.215645 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.215648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.215888 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.216087 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.216091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.216766 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.216768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.217567 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.217569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.217932 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.217934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.218060 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.218589 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.218597 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.219864 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.221364 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.221374 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.221376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.221613 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.221818 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.221818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.221819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.221911 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.222137 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.222140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.222253 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.222328 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.222470 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260928 22:46:55.223201 13819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 22:46:55.230350 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.223441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46578)
0928 22:46:55.223491 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.223502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.223541 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.223760 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.223760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.223896 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.223952 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.223953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.224878 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.224879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.225669 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.225672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.226129 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.226130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.226204 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.226583 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.226589 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.227849 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.229539 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.229540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.229540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.229707 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.229876 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.229877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.229877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.229922 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.230128 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.230128 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.230163 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.230206 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.230282 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:46:55.230564 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.223498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.0.98.61:37245)
0928 22:46:55.223634 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.223638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.223658 (+ 20us) server_negotiation.cc:407] Connection header received
0928 22:46:55.223678 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.223680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.223690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.223747 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.224012 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.224811 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.225753 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.226059 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.226115 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.227929 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.227938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.227938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.227944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.229458 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.229772 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.229775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.229776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.229829 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.229955 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.229955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.229955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.230086 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.230141 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.230229 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.230369 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 22:46:55.231091 13819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:46:55.231400 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.231294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33844)
0928 22:46:55.231327 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.231377 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 22:46:55.231530 13768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
I20260928 22:46:55.231920 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.231825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33852)
0928 22:46:55.231864 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.231904 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:46:55.232620 13819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 22:46:55.245602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35507
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:55.281876 13819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } attempt: 1
I20260928 22:46:55.282351 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.282233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33856)
0928 22:46:55.282285 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.282318 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:46:55.282687 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.282601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33858)
0928 22:46:55.282642 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.282667 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:46:55.282785 13819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
I20260928 22:46:55.293201 13744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } attempt: 3
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:46:55.300144 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.293553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46584)
0928 22:46:55.293620 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.293630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.293677 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.293806 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.293807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.293963 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.294028 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.294030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.294671 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.294673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.295184 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.295186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.295547 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.295548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.295619 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.296091 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.296095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.297611 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.299217 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.299218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.299218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.299372 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.299562 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.299562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.299562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.299606 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.299865 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.299866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.299913 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.299980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.300068 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 22:46:55.300192 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.293626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.98.61:37245)
0928 22:46:55.293685 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.293687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.293713 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:46:55.293740 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.293740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.293752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.293796 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.294098 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.294598 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.295255 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.295486 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.295543 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.297709 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.297712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.297712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.297721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.299144 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.299439 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.299440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.299440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.299505 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.299642 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.299643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.299643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.299775 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.299836 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.300017 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.300098 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:46:55.300624 13744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:46:55.300983 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.300887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33870)
0928 22:46:55.300931 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.300959 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:46:55.301282 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.301199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33882)
0928 22:46:55.301237 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.301260 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:46:55.301395 13744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
W20260928 22:46:55.327963 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:55.328200 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:55.328223 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:55.329521 13826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:55.329576 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:55.329594 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:55.329607 13826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:55.329617 13826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:55.331199 13826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44029,127.0.98.61:37245,127.0.98.60:35507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:55.331444 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:55.331681 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:46:55.332537 13744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } attempt: 1
I20260928 22:46:55.333003 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.332878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33884)
0928 22:46:55.332928 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.332967 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 22:46:55.333349 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.333257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33888)
0928 22:46:55.333300 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.333325 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:46:55.333439 13744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
W20260928 22:46:55.336535 13832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:55.336512 13834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:55.336655 13826 server_base.cc:1034] running on GCE node
W20260928 22:46:55.336535 13831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:55.337001 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:55.337261 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:55.338421 13826 hybrid_clock.cc:648] HybridClock initialized: now 1790635615338411 us; error 32 us; skew 500 ppm
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:55.340868 13826 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 22:46:55.341274 13826 webserver.cc:460] Webserver started at http://127.0.98.60:40237/ using document root <none> and password file <none>
I20260928 22:46:55.341509 13826 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:55.341558 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:55.341675 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:55.342711 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data/instance:
uuid: "037dcc2555fd48bc905148ea1ff0ded0"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "444afd16fc8cf7676171cd4f9eb8732b"
server_key_iv: "cfa6dcd5b05429bb4c7182dd24c1b4af"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.343029 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal/instance:
uuid: "037dcc2555fd48bc905148ea1ff0ded0"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "444afd16fc8cf7676171cd4f9eb8732b"
server_key_iv: "cfa6dcd5b05429bb4c7182dd24c1b4af"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.344218 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:46:55.344904 13841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:55.345048 13826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:55.345121 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "037dcc2555fd48bc905148ea1ff0ded0"
format_stamp: "Formatted at 2026-09-28 22:46:55 on dist-test-slave-pxr1"
server_key: "444afd16fc8cf7676171cd4f9eb8732b"
server_key_iv: "cfa6dcd5b05429bb4c7182dd24c1b4af"
server_key_version: "encryptionkey@0"
I20260928 22:46:55.345180 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:55.350903 13819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } attempt: 2
I20260928 22:46:55.351322 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.351180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33896)
0928 22:46:55.351239 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.351284 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:46:55.351662 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.351560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33910)
0928 22:46:55.351609 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.351639 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 22:46:55.351748 13819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35507: connect: Connection refused (error 111)
I20260928 22:46:55.356465 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:55.358574 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:55.358700 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:55.362367 13826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35507
I20260928 22:46:55.362399 13893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35507 every 8 connection(s)
I20260928 22:46:55.362716 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
I20260928 22:46:55.363520 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:55.365077 13894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:46:55.370278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13826
I20260928 22:46:55.370401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635468897675-392-0/minicluster-data/master-2/wal/instance
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:55.370616 392 external_mini_cluster.cc:1428] Setting key 6e60d73cd6a6dd4d4b5be765b4925901
I20260928 22:46:55.373906 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.365771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.98.62:44029)
0928 22:46:55.365838 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.365840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.366019 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:46:55.366513 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.366515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.366526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.366576 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.367162 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.367635 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.368574 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.368824 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.368889 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.371074 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.371076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.371076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.371085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.372616 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.373042 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.373043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.373044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.373098 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.373337 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.373337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.373337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.373435 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.373478 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.373757 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.373820 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:46:55.374117 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.365721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44029 (local address 127.0.0.1:56922)
0928 22:46:55.365904 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.365939 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.366416 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.366641 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.366644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.366882 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.367073 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.367078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.367712 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.367713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.368496 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.368498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.368895 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.368896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.369014 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.369453 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.369461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.370964 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.372696 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.372702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.372703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.372973 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.373151 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.373151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.373152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.373281 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.373484 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.373489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.373636 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.373743 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.373903 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260928 22:46:55.374663 13894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:46:55.379432 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.370962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.98.60:35507)
0928 22:46:55.371134 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.371139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.371196 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:46:55.371218 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.371220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.371234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.371324 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.371917 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.372950 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.373848 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.374266 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.374332 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.376820 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.376827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.376828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.376834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.378309 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.378663 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.378666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.378667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.378727 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.378903 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.378904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.378904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.379045 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.379190 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.379257 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.379331 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260928 22:46:55.381240 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.375014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.0.98.61:37245)
0928 22:46:55.375059 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.375062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.375070 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:55.375095 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.375095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.375105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.375173 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.375420 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.376018 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.376928 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.377173 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.377243 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.378860 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.378862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.378862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.378870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.380267 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.380536 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.380537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.380537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.380591 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.380795 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.380796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.380796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.380904 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.380956 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.381086 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.381155 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:46:55.381237 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.374920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37245 (local address 127.0.0.1:46596)
0928 22:46:55.374964 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.374974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.375010 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.375175 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.375176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.375280 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.375324 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.375325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.376089 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.376091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.376861 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.376864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.377249 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.377250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.377320 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.377623 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.377628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.378798 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.380322 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.380323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.380323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.380475 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.380644 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.380645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.380645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.380699 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.380956 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.380957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.380998 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.381058 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.381143 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:46:55.381762 13894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:46:55.387440 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.380289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.0.98.62:44029)
0928 22:46:55.380328 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.380330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.380337 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:46:55.380391 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.380392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.380401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.380445 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.380923 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.381456 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.382281 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.382470 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.382527 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.384501 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.384503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.384503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.384509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.385928 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.386239 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.386240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.386241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.386299 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.386467 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.386467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.386468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.386609 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.387227 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.387266 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.387348 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:46:55.387814 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.381983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33926)
0928 22:46:55.382028 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.382044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.382091 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.382196 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.382196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.382337 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.382368 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.382369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.383064 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.383065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.383525 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.383526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.383844 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.383845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.383889 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.384241 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.384246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.385483 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.386946 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.386948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.386948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.387114 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.387294 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.387295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.387295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.387336 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.387564 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.387564 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.387597 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.387645 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.387727 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:46:55.387944 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.382004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.0.98.60:35507)
0928 22:46:55.382044 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.382045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.382079 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:46:55.382139 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.382139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.382146 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.382183 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.382428 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.382988 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.383589 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.383790 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.383840 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.385562 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.385563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.385563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.385570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.386883 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.387177 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.387178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.387179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.387235 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.387369 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.387370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.387370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.387485 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.387543 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.387806 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.387868 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:46:55.389436 13894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: Bootstrap starting.
I20260928 22:46:55.390259 13894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:55.390532 13894 log.cc:824] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:55.391297 13894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: No bootstrap required, opened a new log
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:46:55.392900 13894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.393047 13894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:55.393090 13894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037dcc2555fd48bc905148ea1ff0ded0, State: Initialized, Role: FOLLOWER
I20260928 22:46:55.393220 13894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.393482 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.393546 13894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:55.393565 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:46:55.394804 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.388208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.0.98.61:37245)
0928 22:46:55.388248 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.388250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.388333 (+ 83us) server_negotiation.cc:407] Connection header received
0928 22:46:55.388387 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.388388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.388399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.388456 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.388918 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.389371 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.390038 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.390223 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.390276 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.392345 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.392346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.392347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.392354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.393738 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.394074 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.394076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.394076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.394134 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.394292 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.394293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.394293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.394420 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.394605 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.394646 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.394721 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:46:55.395177 13911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:46:55.395270 13911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:46:55.396248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:46:55.395848 13826 master_runner.cc:186] Error getting master registration for 127.0.98.62:44029: OK, instance_id { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" instance_seqno: 1790635615066316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:46:55.397753 13826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:46:55.405615 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.397760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.98.62:44029)
0928 22:46:55.397802 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.397804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.398066 (+ 262us) server_negotiation.cc:407] Connection header received
0928 22:46:55.398149 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.398150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.398160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.398190 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.398943 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.399712 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.400821 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.401119 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.401162 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.402713 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.402715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.402715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.402723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.404245 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.404665 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.404666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.404667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.404766 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.404968 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.404969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.404969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.405104 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.405179 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.405416 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.405507 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:46:55.405838 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.398420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.0.98.60:35507)
0928 22:46:55.398457 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.398459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.398691 (+ 232us) server_negotiation.cc:407] Connection header received
0928 22:46:55.398782 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.398783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.398792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.398819 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.399551 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.400261 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.401291 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.401593 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.401633 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.403056 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.403058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.403058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.403066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.404519 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.404931 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.404933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.404933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.404991 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.405182 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.405183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.405183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.405357 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.405441 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.405669 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.405749 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:46:55.406333 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.398155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.0.98.61:37245)
0928 22:46:55.398197 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.398199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.398399 (+ 200us) server_negotiation.cc:407] Connection header received
0928 22:46:55.398486 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.398487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.398498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.398525 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.399106 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.399784 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.400820 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.401127 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.401162 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.402886 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.402887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.402887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.402894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.405148 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.405515 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.405517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.405517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.405572 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.405758 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.405759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.405759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.405879 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.405955 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.406155 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.406235 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:46:55.408638 13744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } attempt: 2
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:46:55.415478 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.408909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33952)
0928 22:46:55.408956 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.408967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.409018 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.409183 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.409184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.409374 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.409423 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.409424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.410024 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.410025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.410831 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.410833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.411152 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.411153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.411208 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.411602 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.411607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.412927 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.414593 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.414594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.414594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.414804 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.414977 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.414978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.414978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.415029 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.415260 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.415260 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.415295 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.415339 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.415408 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 22:46:55.415555 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.409045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.60:35507)
0928 22:46:55.409085 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.409087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.409095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:55.409110 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.409111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.409118 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.409160 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.409498 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.409957 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.410910 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.411091 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.411141 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.413009 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.413011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.413011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.413019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.414423 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.414864 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.414865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.414865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.414924 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.415079 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.415079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.415080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.415208 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.415272 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.415375 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.415451 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:46:55.417079 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: Bootstrap starting.
I20260928 22:46:55.417829 13744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:55.418136 13744 log.cc:824] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:55.419210 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: No bootstrap required, opened a new log
I20260928 22:46:55.421247 13744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.421393 13744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:55.421433 13744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d9fb89094749908fd4fa60a848099b, State: Initialized, Role: FOLLOWER
I20260928 22:46:55.421566 13744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.421813 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.421875 13744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:55.421895 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:46:55.423890 13932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:46:55.423997 13932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:46:55.424299 13676 master_runner.cc:186] Error getting master registration for 127.0.98.61:37245: OK, instance_id { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" instance_seqno: 1790635615210367 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:46:55.425966 13676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:46:55.442874 13819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } attempt: 3
Sep 28 22:46:55 dist-test-slave-pxr1 krb5kdc[13663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:46:55.447050 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:46:55.447161 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.447502 13900 leader_election.cc:288] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52d9fb89094749908fd4fa60a848099b (127.0.98.62:44029), ea54ed0a2ca14264932bba6ee8e2add9 (127.0.98.61:37245)
I20260928 22:46:55.447937 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037dcc2555fd48bc905148ea1ff0ded0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d9fb89094749908fd4fa60a848099b" is_pre_election: true
I20260928 22:46:55.448091 13718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 037dcc2555fd48bc905148ea1ff0ded0 in term 0.
I20260928 22:46:55.448331 13845 leader_election.cc:302] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037dcc2555fd48bc905148ea1ff0ded0, 52d9fb89094749908fd4fa60a848099b; no voters:
I20260928 22:46:55.448568 13900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:46:55.448642 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:46:55.448668 13900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:55.449625 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.449748 13900 leader_election.cc:288] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [CANDIDATE]: Term 1 election: Requested vote from peers 52d9fb89094749908fd4fa60a848099b (127.0.98.62:44029), ea54ed0a2ca14264932bba6ee8e2add9 (127.0.98.61:37245)
I20260928 22:46:55.450166 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037dcc2555fd48bc905148ea1ff0ded0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d9fb89094749908fd4fa60a848099b"
I20260928 22:46:55.450248 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.443137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35507 (local address 127.0.0.1:33966)
0928 22:46:55.443188 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:55.443200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:55.443237 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:55.443357 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:55.443357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:55.443507 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:55.443549 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.443550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.444176 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.444177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.444675 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:55.444676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.445015 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.445016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:55.445069 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.445464 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:55.445469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:55.446609 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:55.449029 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.449030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.449031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.449268 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.449516 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:55.449517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:55.449517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:55.449591 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:55.449910 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:55.449911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:55.449978 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 22:46:55.450036 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:46:55.450155 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:46:55.450246 13718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:55.450537 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:55.443188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.60:35507)
0928 22:46:55.443225 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:55.443227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:55.443235 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:55.443295 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:55.443295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:55.443305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:55.443345 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:55.443625 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.444077 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.444772 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:55.444956 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:55.445011 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:55.446687 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:55.446688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:55.446688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:55.446697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:55.448548 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.449365 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.449366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.449366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.449439 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:55.449667 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:55.449668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:55.449669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:55.449813 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:55.449868 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:55.450385 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 22:46:55.450460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:46:55.451203 13718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037dcc2555fd48bc905148ea1ff0ded0 in term 1.
I20260928 22:46:55.451363 13845 leader_election.cc:302] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037dcc2555fd48bc905148ea1ff0ded0, 52d9fb89094749908fd4fa60a848099b; no voters:
I20260928 22:46:55.451655 13900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:55.451798 13900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [term 1 LEADER]: Becoming Leader. State: Replica: 037dcc2555fd48bc905148ea1ff0ded0, State: Running, Role: LEADER
I20260928 22:46:55.451884 13900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.452310 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037dcc2555fd48bc905148ea1ff0ded0. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.452378 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:55.452481 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: Bootstrap starting.
I20260928 22:46:55.452615 13936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:55.452806 13936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:46:55.453548 13819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:55.453883 13819 log.cc:824] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:55.455425 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: No bootstrap required, opened a new log
I20260928 22:46:55.456280 13718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [term 1 FOLLOWER]: Refusing update from remote peer 037dcc2555fd48bc905148ea1ff0ded0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:55.456694 13900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:55.457561 13819 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.457711 13819 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:55.457743 13819 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea54ed0a2ca14264932bba6ee8e2add9, State: Initialized, Role: FOLLOWER
I20260928 22:46:55.457852 13819 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } }
I20260928 22:46:55.458137 13819 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:55.458271 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037dcc2555fd48bc905148ea1ff0ded0. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.458355 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:46:55.458477 13772 tablet.cc:2369] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:46:55.458760 13791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:55.459353 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037dcc2555fd48bc905148ea1ff0ded0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea54ed0a2ca14264932bba6ee8e2add9"
I20260928 22:46:55.459568 13793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037dcc2555fd48bc905148ea1ff0ded0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" is_pre_election: true
I20260928 22:46:55.459661 13791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [term 1 FOLLOWER]: Refusing update from remote peer 037dcc2555fd48bc905148ea1ff0ded0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:55.459918 13900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:46:55.460911 13952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:46:55.461004 13952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:46:55.462195 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.462280 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:55.462265 13937 mvcc.cc:205] Tried to move back new op lower bound from 7334443480905216000 to 7334443480891711488. Current Snapshot: MvccSnapshot[applied={T|T < 7334443480905216000}]
I20260928 22:46:55.462625 13939 mvcc.cc:205] Tried to move back new op lower bound from 7334443480905216000 to 7334443480891711488. Current Snapshot: MvccSnapshot[applied={T|T < 7334443480905216000}]
I20260928 22:46:55.462819 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.462893 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:46:55.463829 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.463912 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:46:55.463960 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037dcc2555fd48bc905148ea1ff0ded0. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.464005 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:46:55.464622 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.464707 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:55.465299 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "037dcc2555fd48bc905148ea1ff0ded0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9fb89094749908fd4fa60a848099b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44029 } } peers { permanent_uuid: "ea54ed0a2ca14264932bba6ee8e2add9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37245 } } peers { permanent_uuid: "037dcc2555fd48bc905148ea1ff0ded0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35507 } } }
I20260928 22:46:55.465373 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:46:55.465477 13955 mvcc.cc:205] Tried to move back new op lower bound from 7334443480905216000 to 7334443480891711488. Current Snapshot: MvccSnapshot[applied={T|T < 7334443480905216000}]
I20260928 22:46:55.469297 13936 catalog_manager.cc:1347] Generated new cluster ID: 735e0b71a9984feb8d6362298fb303bc
I20260928 22:46:55.469411 13936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:55.486267 13936 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:46:55.487177 13936 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:55.497526 13936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 037dcc2555fd48bc905148ea1ff0ded0: Generated new TSK 0
I20260928 22:46:55.497855 13936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:46:56.424592 13932 catalog_manager.cc:1259] Loaded cluster ID: 735e0b71a9984feb8d6362298fb303bc
I20260928 22:46:56.424676 13932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: loading cluster ID for follower catalog manager: success
I20260928 22:46:56.425493 13932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: acquiring CA information for follower catalog manager: success
I20260928 22:46:56.425792 13932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52d9fb89094749908fd4fa60a848099b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:46:56.461530 13952 catalog_manager.cc:1259] Loaded cluster ID: 735e0b71a9984feb8d6362298fb303bc
I20260928 22:46:56.461606 13952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: loading cluster ID for follower catalog manager: success
I20260928 22:46:56.462395 13952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: acquiring CA information for follower catalog manager: success
I20260928 22:46:56.462672 13952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea54ed0a2ca14264932bba6ee8e2add9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:46:56.553153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13676
I20260928 22:46:56.557907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13751
I20260928 22:46:56.561985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13826
2026-09-28T22:46:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2670 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:46:56 dist-test-slave-pxr1 krb5kdc[13966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:46:56 dist-test-slave-pxr1 krb5kdc[13966](info): set up 2 sockets
Sep 28 22:46:56 dist-test-slave-pxr1 krb5kdc[13966](info): commencing operation
krb5kdc: starting...
W20260928 22:46:57.516412 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:46:57.627182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:57.706913 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:57.707109 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:57.707130 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:57.708262 13979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:57.708303 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:57.708317 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:57.708340 13979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:57.708362 13979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:57.709875 13979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:57.710101 13979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:57.710307 13979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:57.714838 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:57.714829 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:57.714829 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:57.715147 13979 server_base.cc:1034] running on GCE node
I20260928 22:46:57.715379 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:57.715649 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:57.716771 13979 hybrid_clock.cc:648] HybridClock initialized: now 1790635617716752 us; error 28 us; skew 500 ppm
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:57.719146 13979 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:57.719544 13979 webserver.cc:460] Webserver started at http://127.0.98.62:44551/ using document root <none> and password file <none>
I20260928 22:46:57.719753 13979 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:57.719800 13979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:57.719929 13979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:57.721000 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "e6056c5aa7714503a85f7b3b4a4788d3"
format_stamp: "Formatted at 2026-09-28 22:46:57 on dist-test-slave-pxr1"
server_key: "e41c8226aa743591edadeab882b7c8a3"
server_key_iv: "12a8e77a5aae7d01250ba82002204151"
server_key_version: "encryptionkey@0"
I20260928 22:46:57.721324 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6056c5aa7714503a85f7b3b4a4788d3"
format_stamp: "Formatted at 2026-09-28 22:46:57 on dist-test-slave-pxr1"
server_key: "e41c8226aa743591edadeab882b7c8a3"
server_key_iv: "12a8e77a5aae7d01250ba82002204151"
server_key_version: "encryptionkey@0"
I20260928 22:46:57.722518 13979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:46:57.723258 13994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:57.723409 13979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:57.723486 13979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "e6056c5aa7714503a85f7b3b4a4788d3"
format_stamp: "Formatted at 2026-09-28 22:46:57 on dist-test-slave-pxr1"
server_key: "e41c8226aa743591edadeab882b7c8a3"
server_key_iv: "12a8e77a5aae7d01250ba82002204151"
server_key_version: "encryptionkey@0"
I20260928 22:46:57.723546 13979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:57.731606 13979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:57.733744 13979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:57.733877 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:57.737512 13979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39861
I20260928 22:46:57.737545 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39861 every 8 connection(s)
I20260928 22:46:57.737880 13979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:57.738694 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:57.740945 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: Bootstrap starting.
I20260928 22:46:57.741439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 13979
I20260928 22:46:57.741499 14047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:57.741537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:46:57.741711 392 external_mini_cluster.cc:1428] Setting key ce36a80c805e1fbbc787c092a89de289
I20260928 22:46:57.741755 14047 log.cc:824] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:57.742472 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: No bootstrap required, opened a new log
I20260928 22:46:57.744683 14047 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260928 22:46:57.744889 14047 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:57.744925 14047 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6056c5aa7714503a85f7b3b4a4788d3, State: Initialized, Role: FOLLOWER
I20260928 22:46:57.745050 14047 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260928 22:46:57.745123 14047 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:57.745157 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:57.745199 14047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:57.745811 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260928 22:46:57.745898 14047 leader_election.cc:302] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6056c5aa7714503a85f7b3b4a4788d3; no voters:
I20260928 22:46:57.746053 14047 leader_election.cc:288] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:46:57.746093 14052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:57.746276 14052 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [term 1 LEADER]: Becoming Leader. State: Replica: e6056c5aa7714503a85f7b3b4a4788d3, State: Running, Role: LEADER
I20260928 22:46:57.746292 14047 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:57.746403 14052 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260928 22:46:57.746763 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } } }
I20260928 22:46:57.746840 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:57.747051 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6056c5aa7714503a85f7b3b4a4788d3. Latest consensus state: current_term: 1 leader_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6056c5aa7714503a85f7b3b4a4788d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } } }
I20260928 22:46:57.747152 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3 [sys.catalog]: This master's current role is: LEADER
W20260928 22:46:57.748021 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:46:57.748077 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:46:57.748122 14059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:57.748281 14059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:57.749758 14059 catalog_manager.cc:1347] Generated new cluster ID: 1b725f318e0347aca683fdc9d88e2fc2
I20260928 22:46:57.749809 14059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:57.752303 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:57.742174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.0.98.62:39861)
0928 22:46:57.742331 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:57.742339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:57.742367 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:46:57.742822 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:57.742830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:57.742879 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:57.742992 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:57.743706 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:57.744570 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:57.745599 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:57.746082 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:57.746192 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:57.748920 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:57.748925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:57.748925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:57.748938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:57.750779 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:57.751153 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:57.751156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:57.751156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:57.751263 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:57.751436 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:57.751436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:57.751437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:57.751740 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:57.751894 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:57.751972 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:46:57.752100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260928 22:46:57.753526 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:46:57.766850 14059 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:46:57.767377 14059 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:57.787335 14059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6056c5aa7714503a85f7b3b4a4788d3: Generated new TSK 0
I20260928 22:46:57.787539 14059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:46:57.807220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 13979
2026-09-28T22:46:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1244 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[14074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[14074](info): set up 2 sockets
Sep 28 22:46:57 dist-test-slave-pxr1 krb5kdc[14074](info): commencing operation
krb5kdc: starting...
W20260928 22:46:58.744410 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:46:58 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:46:58.860576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:58.940893 14087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:58.941099 14087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:58.941121 14087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:58.942291 14087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:58.942335 14087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:58.942348 14087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:58.942360 14087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:58.942373 14087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:58.943876 14087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:58.944106 14087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:58.944330 14087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:58.949062 14092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:58.949048 14095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:58.949150 14087 server_base.cc:1034] running on GCE node
W20260928 22:46:58.949050 14093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:58.949446 14087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:58.949740 14087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:58.950884 14087 hybrid_clock.cc:648] HybridClock initialized: now 1790635618950870 us; error 28 us; skew 500 ppm
Sep 28 22:46:58 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:58.953322 14087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:58.953748 14087 webserver.cc:460] Webserver started at http://127.0.98.62:34525/ using document root <none> and password file <none>
I20260928 22:46:58.953964 14087 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:58.954011 14087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:58.954133 14087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:58.955205 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "0dfff2f701894eec8929d77af854a7b3"
format_stamp: "Formatted at 2026-09-28 22:46:58 on dist-test-slave-pxr1"
server_key: "d697d856b6f7f92872b764bdc7ac49c5"
server_key_iv: "915026a11b0220c4973c78348277aa13"
server_key_version: "encryptionkey@0"
I20260928 22:46:58.955545 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dfff2f701894eec8929d77af854a7b3"
format_stamp: "Formatted at 2026-09-28 22:46:58 on dist-test-slave-pxr1"
server_key: "d697d856b6f7f92872b764bdc7ac49c5"
server_key_iv: "915026a11b0220c4973c78348277aa13"
server_key_version: "encryptionkey@0"
I20260928 22:46:58.956811 14087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:46:58.957487 14102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:58.957638 14087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:46:58.957718 14087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "0dfff2f701894eec8929d77af854a7b3"
format_stamp: "Formatted at 2026-09-28 22:46:58 on dist-test-slave-pxr1"
server_key: "d697d856b6f7f92872b764bdc7ac49c5"
server_key_iv: "915026a11b0220c4973c78348277aa13"
server_key_version: "encryptionkey@0"
I20260928 22:46:58.957787 14087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:58.987345 14087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:58.989367 14087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:58.989486 14087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:58.993221 14087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44945
I20260928 22:46:58.993263 14154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44945 every 8 connection(s)
I20260928 22:46:58.993588 14087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:58.994557 14155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:58.995590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14087
I20260928 22:46:58.995685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:46:58.995862 392 external_mini_cluster.cc:1428] Setting key fcbdf27c9cddd302589d4e97ed8663ef
I20260928 22:46:58.997383 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap starting.
I20260928 22:46:58.998010 14155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:58.998310 14155 log.cc:824] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:58.999023 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: No bootstrap required, opened a new log
I20260928 22:46:59.001186 14155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.001358 14155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:59.001384 14155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.001506 14155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.001575 14155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:59.001597 14155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:59.001643 14155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:59.002202 14155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.002306 14155 leader_election.cc:302] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dfff2f701894eec8929d77af854a7b3; no voters:
I20260928 22:46:59.002472 14155 leader_election.cc:288] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:46:59.002513 14160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 22:46:58 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:59.002657 14160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Running, Role: LEADER
I20260928 22:46:59.002749 14155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:59.002737 14160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.003816 14162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:46:59.003909 14162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:59.004086 14174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:59.004477 14161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dfff2f701894eec8929d77af854a7b3. Latest consensus state: current_term: 1 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:46:59.004554 14161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:59.004638 14174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:46:59.006706 14174 catalog_manager.cc:1347] Generated new cluster ID: 5b6e393f716e425f85113b8d4edb03b6
I20260928 22:46:59.006765 14174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:59.007525 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:58.996369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.62:44945)
0928 22:46:58.996564 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:58.996573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:58.996607 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:46:58.997141 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:58.997151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:58.997214 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:58.997378 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:58.997971 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:58.999256 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:58.999978 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.000614 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.000761 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.003175 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.003180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.003180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.003193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.005370 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.006023 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.006026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.006027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.006155 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.006429 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.006430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.006431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.006745 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.006898 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.007162 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.007320 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 22:46:59.011651 14174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:46:59.012174 14174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:59.020129 14174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Generated new TSK 0
I20260928 22:46:59.020319 14174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:46:59.047204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:59.127204 14181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.127427 14181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.127463 14181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.128669 14181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.128753 14181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.128805 14181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:46:59.130347 14181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.130594 14181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.130828 14181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.135516 14186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.135582 14187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.135511 14189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.135686 14181 server_base.cc:1034] running on GCE node
I20260928 22:46:59.135877 14181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.136076 14181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.137197 14181 hybrid_clock.cc:648] HybridClock initialized: now 1790635619137187 us; error 27 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.139561 14181 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:46:59.139963 14181 webserver.cc:460] Webserver started at http://127.0.98.1:42189/ using document root <none> and password file <none>
I20260928 22:46:59.140170 14181 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.140216 14181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.140344 14181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:59.141407 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa316c0af4c341d3be65e785aaab66a6"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "91f0229cc1920b58879b7a489f80d033"
server_key_iv: "4a8ae9743939dfe66eb2b53161b4f701"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.141750 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa316c0af4c341d3be65e785aaab66a6"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "91f0229cc1920b58879b7a489f80d033"
server_key_iv: "4a8ae9743939dfe66eb2b53161b4f701"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.143023 14181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:46:59.143738 14196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.143886 14181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:46:59.143966 14181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "fa316c0af4c341d3be65e785aaab66a6"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "91f0229cc1920b58879b7a489f80d033"
server_key_iv: "4a8ae9743939dfe66eb2b53161b4f701"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.144025 14181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.152529 14181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.154597 14181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.154731 14181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.154939 14181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:59.155227 14181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:59.155260 14181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.155289 14181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:59.155313 14181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.160744 14181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260928 22:46:59.160792 14309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260928 22:46:59.161098 14181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:46:59.161949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14181
I20260928 22:46:59.162060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:46:59.162254 392 external_mini_cluster.cc:1428] Setting key bbda08b6ebb82172adb15062b5aafa19
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:46:59.170807 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.161800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43173 (local address 127.0.98.62:44945)
0928 22:46:59.162033 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.162035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.162050 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:46:59.162667 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.162668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.162683 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.162735 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.163742 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.164379 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.165449 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.165728 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.165789 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.168000 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.168002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.168002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.168010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.169499 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.169882 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.169883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.169884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.169945 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.170149 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.170150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.170151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.170291 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.170359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.170603 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.170708 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260928 22:46:59.170944 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.161772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:43173)
0928 22:46:59.161937 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.161973 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.162556 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.162811 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.162815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.163429 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.163636 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.163641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.164497 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.164498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.165363 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.165366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.165828 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.165830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.165969 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.166479 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.166488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.167900 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.169578 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.169584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.169586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.169809 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.170006 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.170007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.170007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.170095 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.170347 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.170350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.170468 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.170558 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.170716 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 22:46:59.171415 14310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:46:59.171504 14310 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:59.171695 14310 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:46:59.172169 14119 ts_manager.cc:194] Registered new tserver with Master: fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999)
I20260928 22:46:59.172781 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:46:59.185719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:59.265368 14315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.265580 14315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.265604 14315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.266762 14315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.266803 14315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.266842 14315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:46:59.268291 14315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.268491 14315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.268680 14315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.273494 14321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.273504 14323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.273558 14315 server_base.cc:1034] running on GCE node
W20260928 22:46:59.273509 14320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.273851 14315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.274085 14315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.275218 14315 hybrid_clock.cc:648] HybridClock initialized: now 1790635619275210 us; error 29 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.277657 14315 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:46:59.278069 14315 webserver.cc:460] Webserver started at http://127.0.98.2:36491/ using document root <none> and password file <none>
I20260928 22:46:59.278278 14315 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.278339 14315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.278461 14315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:59.279495 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a55fdbbbe4262001f0d53cb42d934b65"
server_key_iv: "a09288eac9a2f48f50894a9080bcb2a1"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.279826 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a55fdbbbe4262001f0d53cb42d934b65"
server_key_iv: "a09288eac9a2f48f50894a9080bcb2a1"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.281106 14315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:46:59.281826 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.281992 14315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:46:59.282073 14315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a55fdbbbe4262001f0d53cb42d934b65"
server_key_iv: "a09288eac9a2f48f50894a9080bcb2a1"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.282132 14315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.290864 14315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.293287 14315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.293409 14315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.293579 14315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:59.293886 14315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:59.293919 14315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.293947 14315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:59.293964 14315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.299388 14315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36261
I20260928 22:46:59.299463 14443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36261 every 8 connection(s)
I20260928 22:46:59.299741 14315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:46:59.300100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14315
I20260928 22:46:59.300213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:46:59.300397 392 external_mini_cluster.cc:1428] Setting key 8f75f191ce0c0a2bdaff169e07b9614f
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 22:46:59.308868 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.300475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58753 (local address 127.0.98.62:44945)
0928 22:46:59.300540 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.300542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.300775 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:46:59.301321 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.301322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.301333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.301373 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.302403 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.302893 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.304065 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.304263 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.304318 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.306398 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.306399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.306400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.306407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.307668 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.308031 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.308033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.308033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.308084 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.308301 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.308302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.308303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.308429 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.308487 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.308709 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.308802 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:46:59.309067 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.300456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:58753)
0928 22:46:59.300607 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.300648 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.301235 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.301444 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.301448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.302008 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.302318 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.302325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.303032 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.303033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.303987 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.303989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.304331 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.304332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.304494 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.305027 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.305035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.306311 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.307754 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.307761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.307763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.307967 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.308136 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.308137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.308137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.308244 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.308486 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.308491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.308607 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.308689 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.308860 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 22:46:59.309518 14444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:46:59.309604 14444 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 22:46:59.309763 14444 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:46:59.310117 14119 ts_manager.cc:194] Registered new tserver with Master: 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261)
I20260928 22:46:59.310487 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:46:59.323297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:59.402221 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.402441 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.402465 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.403699 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.403739 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.403766 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:46:59.405265 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.405463 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.405649 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.410401 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.410408 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.410440 14449 server_base.cc:1034] running on GCE node
W20260928 22:46:59.410411 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.410704 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.410905 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.412029 14449 hybrid_clock.cc:648] HybridClock initialized: now 1790635619412010 us; error 32 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.414330 14449 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:46:59.414736 14449 webserver.cc:460] Webserver started at http://127.0.98.3:43985/ using document root <none> and password file <none>
I20260928 22:46:59.414925 14449 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.414968 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.415081 14449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:46:59.416078 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "d73814f655b44a8aaa7b68cdb6097e06"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a50783d47090f644776b9a6d5df730ff"
server_key_iv: "40b0464eb07b9f7742a4ccf485fcbced"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.416390 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d73814f655b44a8aaa7b68cdb6097e06"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a50783d47090f644776b9a6d5df730ff"
server_key_iv: "40b0464eb07b9f7742a4ccf485fcbced"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.417650 14449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:46:59.418300 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.418459 14449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:46:59.418542 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "d73814f655b44a8aaa7b68cdb6097e06"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a50783d47090f644776b9a6d5df730ff"
server_key_iv: "40b0464eb07b9f7742a4ccf485fcbced"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.418601 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.439240 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.441279 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.441417 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.441632 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:59.441920 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:59.441953 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.441982 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:59.442006 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.447335 14449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37373
I20260928 22:46:59.447381 14577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37373 every 8 connection(s)
I20260928 22:46:59.447674 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:59.456373 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.448400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37345 (local address 127.0.98.62:44945)
0928 22:46:59.448466 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.448468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.448662 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:46:59.449169 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.449170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.449179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.449231 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.450152 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.450647 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.451605 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.451789 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.451836 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.454044 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.454045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.454046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.454053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.455281 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.455626 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.455627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.455627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.455674 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.455850 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.455851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.455851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.455944 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.455985 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.456243 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.456307 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:46:59.456506 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.448376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:37345)
0928 22:46:59.448543 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.448582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.449083 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.449301 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.449304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.449858 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.450051 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.450085 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.450726 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.450727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.451543 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.451545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.451887 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.451888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.452012 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.452539 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.452548 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.453937 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.455355 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.455361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.455363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.455569 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.455718 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.455718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.455719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.455809 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.455985 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.455987 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.456098 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.456208 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.456329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260928 22:46:59.456939 14578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:46:59.457023 14578 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:59.457203 14578 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:46:59.457544 14119 ts_manager.cc:194] Registered new tserver with Master: d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373)
I20260928 22:46:59.457826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14449
I20260928 22:46:59.457923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:46:59.457913 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37345
I20260928 22:46:59.458127 392 external_mini_cluster.cc:1428] Setting key 8f2da9fe5abadc6e5d41b04777dd1ad5
I20260928 22:46:59.458894 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:46:59.465503 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.460151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.98.62:44945)
0928 22:46:59.460191 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.460193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.460282 (+ 89us) server_negotiation.cc:407] Connection header received
0928 22:46:59.460338 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.460339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.460348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.460392 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.460975 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.461485 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.462195 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.462374 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.462425 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.463478 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.463479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.463479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.463486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.464470 (+ 984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.464793 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.464793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.464794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.464846 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.465005 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.465006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.465006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.465152 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.465242 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.465343 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.465419 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:46:59.466907 14119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:46:59.468432 14119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:46:59.474982 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.470349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36261 (local address 127.0.0.1:55322)
0928 22:46:59.470832 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.470841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.470894 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.471041 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.471042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.471043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:59.471100 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.471102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.472386 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.472387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.473817 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.473820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.474708 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.474709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.474772 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.474783 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.474848 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.474902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":30,"thread_start_us":432,"threads_started":1}
I20260928 22:46:59.475014 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.470440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.98.2:36261)
0928 22:46:59.470616 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.470622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.470907 (+ 285us) server_negotiation.cc:407] Connection header received
0928 22:46:59.470953 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.470955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.470969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.471042 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:59.471150 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.472277 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.473890 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.474625 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.474699 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.474759 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.474889 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.474936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260928 22:46:59.475772 14378 tablet_service.cc:1467] Processing CreateTablet for tablet 48982ebd7d994adbabd7522df8cc6979 (DEFAULT_TABLE table=test-table [id=0856e1566199479e82eef6f9a2cea5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:59.475906 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.470635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.0.98.1:33999)
0928 22:46:59.470786 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.470789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.470797 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:59.470828 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.470830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.470842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.470907 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:59.471168 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.472257 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.474795 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.475521 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.475606 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.475660 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.475782 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.475830 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260928 22:46:59.476038 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.470896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37373 (local address 127.0.0.1:58300)
0928 22:46:59.470994 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.471001 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.471037 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.472818 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.472819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.472821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:59.472875 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.472876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.473822 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.473823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.474877 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.474880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.475818 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.475819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.475874 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.475877 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.475915 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.475951 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260928 22:46:59.476118 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.470349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:46318)
0928 22:46:59.470473 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.470502 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.470555 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.470998 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.471001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.471003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:59.471091 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.471093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.472386 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.472387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.473817 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.473820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.475608 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.475609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.475672 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.475678 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.476004 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.476054 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260928 22:46:59.476161 14378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48982ebd7d994adbabd7522df8cc6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:59.476332 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.472472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.98.3:37373)
0928 22:46:59.472630 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.472634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.472642 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:59.472671 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.472673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.472685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.472808 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:59.472928 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.473724 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.474962 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.475675 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.475734 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.475785 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.476215 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.476260 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 22:46:59.476778 14244 tablet_service.cc:1467] Processing CreateTablet for tablet 48982ebd7d994adbabd7522df8cc6979 (DEFAULT_TABLE table=test-table [id=0856e1566199479e82eef6f9a2cea5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:59.477034 14244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48982ebd7d994adbabd7522df8cc6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:59.476981 14512 tablet_service.cc:1467] Processing CreateTablet for tablet 48982ebd7d994adbabd7522df8cc6979 (DEFAULT_TABLE table=test-table [id=0856e1566199479e82eef6f9a2cea5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:46:59.477303 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48982ebd7d994adbabd7522df8cc6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:46:59.479341 14592 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Bootstrap starting.
I20260928 22:46:59.479908 14593 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Bootstrap starting.
I20260928 22:46:59.480012 14592 tablet_bootstrap.cc:654] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:59.480381 14594 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Bootstrap starting.
I20260928 22:46:59.480572 14592 log.cc:824] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:59.480614 14593 tablet_bootstrap.cc:654] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:59.481033 14594 tablet_bootstrap.cc:654] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Neither blocks nor log segments found. Creating new log.
I20260928 22:46:59.481096 14593 log.cc:824] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:59.481401 14592 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: No bootstrap required, opened a new log
I20260928 22:46:59.481465 14592 ts_tablet_manager.cc:1397] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:46:59.481504 14594 log.cc:824] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:59.481871 14593 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: No bootstrap required, opened a new log
I20260928 22:46:59.481941 14593 ts_tablet_manager.cc:1397] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:46:59.482155 14594 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: No bootstrap required, opened a new log
I20260928 22:46:59.482211 14594 ts_tablet_manager.cc:1397] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:46:59.483417 14592 raft_consensus.cc:348] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.483604 14592 raft_consensus.cc:374] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:59.483640 14592 raft_consensus.cc:729] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa316c0af4c341d3be65e785aaab66a6, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.483793 14592 consensus_queue.cc:260] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.483810 14594 raft_consensus.cc:348] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.483824 14593 raft_consensus.cc:348] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.483953 14594 raft_consensus.cc:374] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:59.483971 14593 raft_consensus.cc:374] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:46:59.483984 14594 raft_consensus.cc:729] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73814f655b44a8aaa7b68cdb6097e06, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.484010 14593 raft_consensus.cc:729] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e80fa9a88c4830a08b7b0acc2f1bea, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.484100 14592 ts_tablet_manager.cc:1428] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:46:59.484099 14594 consensus_queue.cc:260] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.484156 14593 consensus_queue.cc:260] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.484217 14310 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
I20260928 22:46:59.484351 14594 ts_tablet_manager.cc:1428] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:46:59.484402 14593 ts_tablet_manager.cc:1428] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:46:59.484452 14578 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
I20260928 22:46:59.484459 14444 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
I20260928 22:46:59.517424 14599 raft_consensus.cc:482] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:46:59.517552 14599 raft_consensus.cc:504] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.517921 14599 leader_election.cc:288] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999), 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261)
I20260928 22:46:59.520666 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.518076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36261 (local address 127.0.98.3:53711)
0928 22:46:59.518133 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.518145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.518193 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.518370 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.518371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.518373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:59.518438 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.518440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.519023 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.519024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.519896 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.519898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.520407 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.520407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.520488 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.520509 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.520554 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.520595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:46:59.520774 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.518165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57219 (local address 127.0.98.1:33999)
0928 22:46:59.518221 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.518223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.518357 (+ 134us) server_negotiation.cc:407] Connection header received
0928 22:46:59.518406 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.518407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.518419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.518462 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:59.518590 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.519040 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.520088 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.520459 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.520518 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.520528 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.520653 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.520691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:46:59.520803 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.518076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.98.3:57219)
0928 22:46:59.518289 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.518299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.518354 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.518473 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.518474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.518475 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:46:59.518539 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.518541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.519107 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.519108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.520016 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.520019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.520530 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.520530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.520600 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.520604 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.520649 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.520685 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260928 22:46:59.520985 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.518221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53711 (local address 127.0.98.2:36261)
0928 22:46:59.518273 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.518275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.518284 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:46:59.518307 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.518308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.518317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.518361 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:46:59.518494 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.518957 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.519968 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.520341 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.520402 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.520413 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.520874 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.520922 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:46:59.521137 14264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48982ebd7d994adbabd7522df8cc6979" candidate_uuid: "d73814f655b44a8aaa7b68cdb6097e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa316c0af4c341d3be65e785aaab66a6" is_pre_election: true
I20260928 22:46:59.521298 14264 raft_consensus.cc:2454] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73814f655b44a8aaa7b68cdb6097e06 in term 0.
I20260928 22:46:59.521250 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48982ebd7d994adbabd7522df8cc6979" candidate_uuid: "d73814f655b44a8aaa7b68cdb6097e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" is_pre_election: true
I20260928 22:46:59.521401 14398 raft_consensus.cc:2454] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73814f655b44a8aaa7b68cdb6097e06 in term 0.
I20260928 22:46:59.521497 14468 leader_election.cc:302] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d73814f655b44a8aaa7b68cdb6097e06, fa316c0af4c341d3be65e785aaab66a6; no voters:
I20260928 22:46:59.521646 14599 raft_consensus.cc:2781] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:46:59.521710 14599 raft_consensus.cc:482] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:46:59.521739 14599 raft_consensus.cc:3037] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:59.522433 14599 raft_consensus.cc:504] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.522557 14599 leader_election.cc:288] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [CANDIDATE]: Term 1 election: Requested vote from peers fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999), 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261)
I20260928 22:46:59.522719 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48982ebd7d994adbabd7522df8cc6979" candidate_uuid: "d73814f655b44a8aaa7b68cdb6097e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
I20260928 22:46:59.522747 14264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48982ebd7d994adbabd7522df8cc6979" candidate_uuid: "d73814f655b44a8aaa7b68cdb6097e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa316c0af4c341d3be65e785aaab66a6"
I20260928 22:46:59.522791 14398 raft_consensus.cc:3037] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:59.522810 14264 raft_consensus.cc:3037] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:46:59.523483 14398 raft_consensus.cc:2454] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73814f655b44a8aaa7b68cdb6097e06 in term 1.
I20260928 22:46:59.523505 14264 raft_consensus.cc:2454] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73814f655b44a8aaa7b68cdb6097e06 in term 1.
I20260928 22:46:59.523645 14466 leader_election.cc:302] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e80fa9a88c4830a08b7b0acc2f1bea, d73814f655b44a8aaa7b68cdb6097e06; no voters:
I20260928 22:46:59.523751 14599 raft_consensus.cc:2781] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:46:59.523869 14599 raft_consensus.cc:686] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 LEADER]: Becoming Leader. State: Replica: d73814f655b44a8aaa7b68cdb6097e06, State: Running, Role: LEADER
I20260928 22:46:59.523959 14599 consensus_queue.cc:237] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.524778 14118 catalog_manager.cc:5495] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d73814f655b44a8aaa7b68cdb6097e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } health_report { overall_health: HEALTHY } } }
I20260928 22:46:59.535730 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.533518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.0.98.3:37373)
0928 22:46:59.533575 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.533577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.533585 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:46:59.533623 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.533624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.533643 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.533701 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:46:59.533822 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.534281 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.535028 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.535214 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.535268 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.535407 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:46:59.535509 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:46:59.535581 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.535601 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.535621 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:46:59.537428 14264 raft_consensus.cc:1261] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Refusing update from remote peer d73814f655b44a8aaa7b68cdb6097e06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:59.537433 14398 raft_consensus.cc:1261] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Refusing update from remote peer d73814f655b44a8aaa7b68cdb6097e06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:46:59.537783 14602 consensus_queue.cc:1035] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:59.537918 14599 consensus_queue.cc:1035] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:46:59.540064 14605 mvcc.cc:205] Tried to move back new op lower bound from 7334443497622953984 to 7334443497570811904. Current Snapshot: MvccSnapshot[applied={T|T < 7334443497622953984}]
I20260928 22:46:59.540848 14607 mvcc.cc:205] Tried to move back new op lower bound from 7334443497622953984 to 7334443497570811904. Current Snapshot: MvccSnapshot[applied={T|T < 7334443497622953984 or (T in {7334443497622953984})}]
I20260928 22:46:59.540967 14606 mvcc.cc:205] Tried to move back new op lower bound from 7334443497622953984 to 7334443497570811904. Current Snapshot: MvccSnapshot[applied={T|T < 7334443497622953984}]
I20260928 22:46:59.542451 14118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:59.542562 14118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:46:59.543275 14118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Sending DeleteTablet for 3 replicas of tablet 48982ebd7d994adbabd7522df8cc6979
I20260928 22:46:59.543612 14512 tablet_service.cc:1514] Processing DeleteTablet for tablet 48982ebd7d994adbabd7522df8cc6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58300
I20260928 22:46:59.543623 14244 tablet_service.cc:1514] Processing DeleteTablet for tablet 48982ebd7d994adbabd7522df8cc6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46318
I20260928 22:46:59.543663 14378 tablet_service.cc:1514] Processing DeleteTablet for tablet 48982ebd7d994adbabd7522df8cc6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55322
I20260928 22:46:59.543841 14617 tablet_replica.cc:330] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: stopping tablet replica
I20260928 22:46:59.543845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14181
I20260928 22:46:59.543937 14617 raft_consensus.cc:2229] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:46:59.544032 14617 raft_consensus.cc:2258] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:59.544153 14615 tablet_replica.cc:330] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: stopping tablet replica
I20260928 22:46:59.544221 14615 raft_consensus.cc:2229] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:46:59.544440 14615 raft_consensus.cc:2258] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:59.544966 14615 ts_tablet_manager.cc:1905] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:46:59.546377 14617 ts_tablet_manager.cc:1905] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:46:59.547556 14615 ts_tablet_manager.cc:1918] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:46:59.547631 14615 log.cc:1196] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/48982ebd7d994adbabd7522df8cc6979
I20260928 22:46:59.547874 14615 ts_tablet_manager.cc:1939] T 48982ebd7d994adbabd7522df8cc6979 P d73814f655b44a8aaa7b68cdb6097e06: Deleting consensus metadata
I20260928 22:46:59.548244 14617 ts_tablet_manager.cc:1918] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:46:59.548327 14617 log.cc:1196] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/48982ebd7d994adbabd7522df8cc6979
I20260928 22:46:59.548305 14104 catalog_manager.cc:4841] TS d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373): tablet 48982ebd7d994adbabd7522df8cc6979 (table test-table [id=0856e1566199479e82eef6f9a2cea5a7]) successfully deleted
I20260928 22:46:59.548566 14617 ts_tablet_manager.cc:1939] T 48982ebd7d994adbabd7522df8cc6979 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting consensus metadata
I20260928 22:46:59.548955 14104 catalog_manager.cc:4841] TS 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261): tablet 48982ebd7d994adbabd7522df8cc6979 (table test-table [id=0856e1566199479e82eef6f9a2cea5a7]) successfully deleted
W20260928 22:46:59.551043 14106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33999 (error 108)
I20260928 22:46:59.551290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14315
I20260928 22:46:59.551460 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.551343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:46324)
0928 22:46:59.551397 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.551427 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:46:59.551554 14106 catalog_manager.cc:4584] TS fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48982ebd7d994adbabd7522df8cc6979: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
I20260928 22:46:59.556514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14449
I20260928 22:46:59.560788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14087
I20260928 22:46:59.565362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--webserver_interface=127.0.98.62
--webserver_port=34525
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:46:59.644236 14619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.644447 14619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.644469 14619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.645681 14619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:46:59.645728 14619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.645742 14619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.645756 14619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:46:59.645778 14619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:46:59.647291 14619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.647526 14619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.647759 14619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.655905 14626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.656039 14619 server_base.cc:1034] running on GCE node
W20260928 22:46:59.655923 14625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.655946 14628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.656392 14619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.656699 14619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.657871 14619 hybrid_clock.cc:648] HybridClock initialized: now 1790635619657853 us; error 33 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.660329 14619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:46:59.660773 14619 webserver.cc:460] Webserver started at http://127.0.98.62:34525/ using document root <none> and password file <none>
I20260928 22:46:59.660990 14619 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.661051 14619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.662359 14619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:46:59.662935 14635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.663098 14619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:46:59.663185 14619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "0dfff2f701894eec8929d77af854a7b3"
format_stamp: "Formatted at 2026-09-28 22:46:58 on dist-test-slave-pxr1"
server_key: "d697d856b6f7f92872b764bdc7ac49c5"
server_key_iv: "915026a11b0220c4973c78348277aa13"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.663576 14619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.688961 14619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.690901 14619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.691043 14619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.694834 14619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44945
I20260928 22:46:59.694875 14687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44945 every 8 connection(s)
I20260928 22:46:59.695245 14619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:46:59.696868 14688 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:46:59.697515 14688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap starting.
I20260928 22:46:59.699853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14619
I20260928 22:46:59.700114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33999
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=42189
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:59.705834 14688 log.cc:824] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:59.707795 14688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:46:59.708050 14688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap complete.
I20260928 22:46:59.710199 14688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.710430 14688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.710574 14688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.710639 14688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:46:59.710675 14688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:46:59.710728 14688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:46:59.711763 14688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.711859 14688 leader_election.cc:302] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dfff2f701894eec8929d77af854a7b3; no voters:
I20260928 22:46:59.712031 14688 leader_election.cc:288] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:46:59.712085 14692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:46:59.712209 14692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 LEADER]: Becoming Leader. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Running, Role: LEADER
I20260928 22:46:59.712297 14688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:46:59.712306 14692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:46:59.712632 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dfff2f701894eec8929d77af854a7b3. Latest consensus state: current_term: 2 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:46:59.712709 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:59.712639 14694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:46:59.712944 14694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:46:59.713131 14696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:46:59.713977 14696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:46:59.714084 14696 catalog_manager.cc:1259] Loaded cluster ID: 5b6e393f716e425f85113b8d4edb03b6
I20260928 22:46:59.714116 14696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:46:59.714346 14696 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 22:46:59.718930 14696 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 22:46:59.721536 14696 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:46:59.721769 14696 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Loaded TSK: 0
I20260928 22:46:59.721992 14696 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:46:59.781903 14690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.782131 14690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.782163 14690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.783501 14690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.783564 14690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.783605 14690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:46:59.785287 14690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42189
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.785518 14690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.785734 14690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.790416 14715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.790429 14716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.790529 14718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.790776 14690 server_base.cc:1034] running on GCE node
I20260928 22:46:59.790992 14690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.791210 14690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.792383 14690 hybrid_clock.cc:648] HybridClock initialized: now 1790635619792367 us; error 31 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.794996 14690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:46:59.795457 14690 webserver.cc:460] Webserver started at http://127.0.98.1:42189/ using document root <none> and password file <none>
I20260928 22:46:59.795666 14690 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.795727 14690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.797086 14690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:46:59.797730 14725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.797889 14690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:46:59.797974 14690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "fa316c0af4c341d3be65e785aaab66a6"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "91f0229cc1920b58879b7a489f80d033"
server_key_iv: "4a8ae9743939dfe66eb2b53161b4f701"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.798338 14690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.808887 14690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.810878 14690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.810998 14690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.811225 14690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:59.811667 14732 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:46:59.813148 14690 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:46:59.813218 14690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:46:59.813263 14690 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:46:59.813880 14690 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:46:59.813920 14690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:46:59.813984 14732 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Bootstrap starting.
I20260928 22:46:59.821373 14690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260928 22:46:59.821902 14690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:46:59.822480 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260928 22:46:59.824851 14732 log.cc:824] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:46:59.825155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14690
I20260928 22:46:59.825460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36261
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=36491
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:59.827509 14732 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:46:59.827806 14732 tablet_bootstrap.cc:492] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Bootstrap complete.
I20260928 22:46:59.828877 14732 ts_tablet_manager.cc:1397] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.013s
I20260928 22:46:59.830163 14732 raft_consensus.cc:348] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
I20260928 22:46:59.830490 14732 raft_consensus.cc:729] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa316c0af4c341d3be65e785aaab66a6, State: Initialized, Role: FOLLOWER
I20260928 22:46:59.830647 14732 consensus_queue.cc:260] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } }
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:59.830920 14732 ts_tablet_manager.cc:1428] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:46:59.835377 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.822617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:60533)
0928 22:46:59.823014 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.823060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.823703 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.824402 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.824407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.824823 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.825105 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.825113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.826742 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.826744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.827964 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.827967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.828770 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.828771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.828925 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.829570 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.829580 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.831359 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.833552 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.833559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.833561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.833849 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.834154 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.834155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.834155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.834306 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.834758 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.834765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.834910 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.834995 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.835162 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":86,"threads_started":1}
I20260928 22:46:59.836329 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60533 (local address 127.0.98.62:44945)
0928 22:46:59.823913 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.823923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.823965 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:46:59.824096 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.824102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.824159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.824333 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.825254 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.826609 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.828075 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.828652 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.828768 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.831490 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.831494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.831494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.831506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.833451 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.833953 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.833957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.833957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.834073 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.834382 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.834383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.834383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.834678 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.835908 (+ 1230us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.835995 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.836129 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":96,"threads_started":1}
I20260928 22:46:59.837275 14840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:46:59.837373 14840 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:59.837565 14840 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:46:59.838369 14652 ts_manager.cc:194] Registered new tserver with Master: fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999)
I20260928 22:46:59.839537 14652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60533
I20260928 22:46:59.840538 14840 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 22:46:59.848338 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.838969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:46336)
0928 22:46:59.839439 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.839474 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.839569 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.839809 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.839812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.840133 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.840191 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.840193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.841529 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.841530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.842493 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.842496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.843148 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.843151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.843228 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.843784 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.843790 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.845395 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.847342 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.847347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.847348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.847536 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.847730 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.847730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.847730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.847780 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.848061 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.848065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.848107 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.848164 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.848265 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":92,"threads_started":1}
I20260928 22:46:59.848382 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.839212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.98.1:33999)
0928 22:46:59.839367 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.839371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.839531 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:46:59.839659 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.839662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.839676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.839714 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.840295 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.841423 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.842585 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.843065 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.843144 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.845487 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.845497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.845498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.845506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.847257 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.847606 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.847609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.847609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.847673 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.847828 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.847829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.847829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.848007 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.848086 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.848196 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.848292 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 22:46:59.848820 14774 tablet_service.cc:1514] Processing DeleteTablet for tablet 48982ebd7d994adbabd7522df8cc6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46336
I20260928 22:46:59.849421 14849 tablet_replica.cc:330] stopping tablet replica
I20260928 22:46:59.849514 14849 raft_consensus.cc:2229] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:46:59.849577 14849 raft_consensus.cc:2258] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:46:59.849860 14849 ts_tablet_manager.cc:1905] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:46:59.851855 14849 ts_tablet_manager.cc:1918] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:46:59.851917 14849 log.cc:1196] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/48982ebd7d994adbabd7522df8cc6979
I20260928 22:46:59.852111 14849 ts_tablet_manager.cc:1939] T 48982ebd7d994adbabd7522df8cc6979 P fa316c0af4c341d3be65e785aaab66a6: Deleting consensus metadata
I20260928 22:46:59.852566 14639 catalog_manager.cc:4841] TS fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999): tablet 48982ebd7d994adbabd7522df8cc6979 (table test-table [id=0856e1566199479e82eef6f9a2cea5a7]) successfully deleted
W20260928 22:46:59.910737 14845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:46:59.910949 14845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:46:59.910980 14845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:46:59.912204 14845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:46:59.912253 14845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:46:59.912297 14845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:46:59.913853 14845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36491
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:46:59.914108 14845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:46:59.914342 14845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:46:59.919093 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:46:59.919107 14854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.919315 14845 server_base.cc:1034] running on GCE node
W20260928 22:46:59.919315 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:46:59.919564 14845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:46:59.919809 14845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:46:59.920945 14845 hybrid_clock.cc:648] HybridClock initialized: now 1790635619920934 us; error 24 us; skew 500 ppm
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:46:59.923282 14845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:46:59.923723 14845 webserver.cc:460] Webserver started at http://127.0.98.2:36491/ using document root <none> and password file <none>
I20260928 22:46:59.923923 14845 fs_manager.cc:359] Metadata directory not provided
I20260928 22:46:59.923983 14845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:46:59.925788 14845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:59.926471 14864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.926661 14845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:46:59.926796 14845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a55fdbbbe4262001f0d53cb42d934b65"
server_key_iv: "a09288eac9a2f48f50894a9080bcb2a1"
server_key_version: "encryptionkey@0"
I20260928 22:46:59.927251 14845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:46:59.935186 14845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:46:59.937233 14845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:46:59.937358 14845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:46:59.937556 14845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:46:59.937825 14845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:46:59.937856 14845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.937878 14845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:46:59.937901 14845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:46:59.943197 14845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36261
I20260928 22:46:59.943260 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36261 every 8 connection(s)
I20260928 22:46:59.943555 14845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:46:59 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:46:59.951931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14845
I20260928 22:46:59.952131 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.944197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49407 (local address 127.0.98.62:44945)
0928 22:46:59.944245 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:46:59.944247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:46:59.944449 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:46:59.944957 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:46:59.944957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:46:59.944973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:46:59.945033 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:46:59.945930 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.946427 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.947364 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.947613 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.947680 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.949636 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:46:59.949638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:46:59.949638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:46:59.949645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:46:59.950882 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.951254 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.951256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.951256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.951309 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:46:59.951526 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:46:59.951527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:46:59.951528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:46:59.951636 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:46:59.951693 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:46:59.951949 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 22:46:59.952041 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:46:59.952175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37373
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=43985
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:46:59.952303 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:46:59.944177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:49407)
0928 22:46:59.944334 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:46:59.944377 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:46:59.944874 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:46:59.945100 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:46:59.945103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:46:59.945663 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:46:59.945855 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.945860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.946493 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.946494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.947290 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:46:59.947292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:46:59.947682 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:46:59.947683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:46:59.947782 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:46:59.948203 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:46:59.948211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:46:59.949538 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:46:59.950962 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.950971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.950973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.951186 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.951363 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:46:59.951364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:46:59.951364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:46:59.951470 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:46:59.951690 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:46:59.951696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:46:59.951835 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 22:46:59.951922 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:46:59.952079 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260928 22:46:59.952736 14978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:46:59.952843 14978 heartbeater.cc:464] Registering TS with master...
I20260928 22:46:59.953032 14978 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:46:59.953405 14652 ts_manager.cc:194] Registered new tserver with Master: 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261)
I20260928 22:46:59.953920 14652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49407
W20260928 22:47:00.032651 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:00.032882 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:00.032907 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:00.034077 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:00.034117 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:00.034143 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:47:00.035585 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43985
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:00.035794 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:00.036087 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:00.040982 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.041006 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.041014 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.041638 14981 server_base.cc:1034] running on GCE node
I20260928 22:47:00.041806 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:00.042016 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:00.043149 14981 hybrid_clock.cc:648] HybridClock initialized: now 1790635620043136 us; error 28 us; skew 500 ppm
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:00.048866 14981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:47:00.049314 14981 webserver.cc:460] Webserver started at http://127.0.98.3:43985/ using document root <none> and password file <none>
I20260928 22:47:00.049531 14981 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:00.049592 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:00.050941 14981 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:00.051465 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.051604 14981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:47:00.051687 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "d73814f655b44a8aaa7b68cdb6097e06"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a50783d47090f644776b9a6d5df730ff"
server_key_iv: "40b0464eb07b9f7742a4ccf485fcbced"
server_key_version: "encryptionkey@0"
I20260928 22:47:00.052033 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:00.060067 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:00.062114 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:00.062249 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:00.062469 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:00.062758 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:00.062791 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.062821 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:00.062844 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.068568 14981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37373
I20260928 22:47:00.068637 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37373 every 8 connection(s)
I20260928 22:47:00.069010 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:00.077135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 14981
I20260928 22:47:00.077703 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.069719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58013 (local address 127.0.98.62:44945)
0928 22:47:00.069765 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.069767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.069975 (+ 208us) server_negotiation.cc:407] Connection header received
0928 22:47:00.070476 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.070478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.070491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.070541 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:00.071477 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.071965 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.073052 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.073250 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.073314 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.075267 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:00.075268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:00.075269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:00.075277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:00.076381 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:00.076753 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:00.076755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:00.076755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:00.076809 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:00.077047 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:00.077048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:00.077048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:00.077160 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:00.077216 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.077523 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.077620 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:47:00.077963 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.069714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:58013)
0928 22:47:00.069853 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.069889 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.070377 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.070613 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:00.070617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:00.071182 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:00.071381 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.071386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.072047 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.072048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.072973 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.072976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.073324 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.073325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.073443 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.073875 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:00.073884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:00.075172 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:00.076452 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:00.076459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:00.076461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:00.076663 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:00.076864 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:00.076865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:00.076865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:00.076984 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:00.077214 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:00.077220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:00.077372 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 22:47:00.077480 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 22:47:00.077671 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 22:47:00.078400 15110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44945
I20260928 22:47:00.078549 15110 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:00.078745 15110 heartbeater.cc:511] Master 127.0.98.62:44945 requested a full tablet report, sending...
I20260928 22:47:00.079133 14652 ts_manager.cc:194] Registered new tserver with Master: d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373)
I20260928 22:47:00.079502 14652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58013
I20260928 22:47:00.083432 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.077498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.98.62:44945)
0928 22:47:00.077611 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.077613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.077701 (+ 88us) server_negotiation.cc:407] Connection header received
0928 22:47:00.077843 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.077844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.077853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.077919 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:00.078537 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.079180 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.079985 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.080173 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.080224 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.081358 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:00.081359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:00.081359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:00.081366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:00.082494 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:00.082844 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:00.082844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:00.082844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:00.082892 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:00.083023 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:00.083024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:00.083024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:00.083119 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:00.083259 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.083286 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.083350 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260928 22:47:00.084079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:47:00.086665 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.084388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.98.62:44945)
0928 22:47:00.084427 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.084429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.084511 (+ 82us) server_negotiation.cc:407] Connection header received
0928 22:47:00.084572 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.084573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.084603 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.084667 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:47:00.084800 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.085281 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.085985 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.086180 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.086273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.086414 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:47:00.086513 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:47:00.086569 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.086585 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.086605 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:47:00.086901 14652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:47:00.088503 14652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:47:00.090648 14774 tablet_service.cc:1467] Processing CreateTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 (DEFAULT_TABLE table=test-table [id=28e33e0294504039967b5430ae3478cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:00.090842 14774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44d3fac3b54447e8e66b7fd3ad32046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:00.092381 14732 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Bootstrap starting.
I20260928 22:47:00.092680 14732 tablet_bootstrap.cc:654] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:00.093163 14732 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: No bootstrap required, opened a new log
I20260928 22:47:00.093196 14732 ts_tablet_manager.cc:1397] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:00.093298 14732 raft_consensus.cc:348] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.093375 14732 raft_consensus.cc:374] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:00.093420 14732 raft_consensus.cc:729] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa316c0af4c341d3be65e785aaab66a6, State: Initialized, Role: FOLLOWER
I20260928 22:47:00.093464 14732 consensus_queue.cc:260] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.093551 14732 ts_tablet_manager.cc:1428] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.094342 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.090346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.0.98.2:36261)
0928 22:47:00.090507 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.090510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.090518 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:47:00.090550 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.090552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.090564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.090937 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:00.091010 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.091886 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.093258 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.093992 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.094059 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.094122 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.094243 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.094278 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260928 22:47:00.094338 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.090250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36261 (local address 127.0.0.1:55338)
0928 22:47:00.090284 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.090295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.090328 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.090823 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:00.090823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:00.090825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:00.090871 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.090873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.091981 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.091982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.093171 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.093174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.094071 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.094072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.094152 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.094172 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 22:47:00.094219 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:47:00.094266 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 22:47:00.094466 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.090343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37373 (local address 127.0.0.1:58306)
0928 22:47:00.090687 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.090695 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.090745 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.090956 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:00.090957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:00.090959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:00.090991 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.090992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.092238 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.092239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.093287 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.093289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.094207 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.094208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.094260 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.094263 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:47:00.094304 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 22:47:00.094333 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":94,"threads_started":1}
I20260928 22:47:00.094492 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.090499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.98.3:37373)
0928 22:47:00.090661 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.090664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.090753 (+ 89us) server_negotiation.cc:407] Connection header received
0928 22:47:00.090802 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.090805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.090821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.090904 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:00.091059 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.092132 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.093368 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.094061 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.094123 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.094176 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.094376 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.094417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260928 22:47:00.095070 14912 tablet_service.cc:1467] Processing CreateTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 (DEFAULT_TABLE table=test-table [id=28e33e0294504039967b5430ae3478cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:00.095067 15044 tablet_service.cc:1467] Processing CreateTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 (DEFAULT_TABLE table=test-table [id=28e33e0294504039967b5430ae3478cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:00.095407 14912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44d3fac3b54447e8e66b7fd3ad32046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:00.095407 15044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44d3fac3b54447e8e66b7fd3ad32046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:00.099040 15120 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Bootstrap starting.
I20260928 22:47:00.099661 15120 tablet_bootstrap.cc:654] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:00.100113 15120 log.cc:824] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:00.100399 15121 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Bootstrap starting.
I20260928 22:47:00.100869 15120 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: No bootstrap required, opened a new log
I20260928 22:47:00.100926 15120 ts_tablet_manager.cc:1397] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:47:00.100962 15121 tablet_bootstrap.cc:654] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:00.101377 15121 log.cc:824] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:00.102037 15121 tablet_bootstrap.cc:492] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: No bootstrap required, opened a new log
I20260928 22:47:00.102103 15121 ts_tablet_manager.cc:1397] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:47:00.102559 15120 raft_consensus.cc:348] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.102720 15120 raft_consensus.cc:374] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:00.102759 15120 raft_consensus.cc:729] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e80fa9a88c4830a08b7b0acc2f1bea, State: Initialized, Role: FOLLOWER
I20260928 22:47:00.102872 15120 consensus_queue.cc:260] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.103119 15120 ts_tablet_manager.cc:1428] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:47:00.103184 14978 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
I20260928 22:47:00.103819 15121 raft_consensus.cc:348] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.103962 15121 raft_consensus.cc:374] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:00.104003 15121 raft_consensus.cc:729] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73814f655b44a8aaa7b68cdb6097e06, State: Initialized, Role: FOLLOWER
I20260928 22:47:00.104115 15121 consensus_queue.cc:260] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.104358 15121 ts_tablet_manager.cc:1428] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:47:00.104391 15110 heartbeater.cc:503] Master 127.0.98.62:44945 was elected leader, sending a full tablet report...
W20260928 22:47:00.194069 14979 tablet.cc:2369] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:47:00.206365 15124 raft_consensus.cc:482] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:47:00.206508 15124 raft_consensus.cc:504] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.206883 15124 leader_election.cc:288] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373), fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999)
I20260928 22:47:00.209585 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.207056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37373 (local address 127.0.98.2:41625)
0928 22:47:00.207104 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.207116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.207161 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.207311 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:00.207312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:00.207314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:00.207383 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.207384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.207985 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.207985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.208821 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.208823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.209349 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.209349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.209423 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.209427 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:47:00.209471 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:47:00.209511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:47:00.209585 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.207122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41625 (local address 127.0.98.3:37373)
0928 22:47:00.207173 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.207174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.207218 (+ 44us) server_negotiation.cc:407] Connection header received
0928 22:47:00.207244 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.207244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.207256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.207295 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:00.207440 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.207908 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.208897 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.209279 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.209338 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.209348 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.209481 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.209520 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:47:00.209893 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44d3fac3b54447e8e66b7fd3ad32046" candidate_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73814f655b44a8aaa7b68cdb6097e06" is_pre_election: true
I20260928 22:47:00.210059 15064 raft_consensus.cc:2454] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e80fa9a88c4830a08b7b0acc2f1bea in term 0.
I20260928 22:47:00.210206 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.207057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.98.2:57763)
0928 22:47:00.207301 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.207311 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.207370 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.207507 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:00.207507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:00.207509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:00.207595 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.207597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.208499 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.208500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.209274 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:00.209276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.209926 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.209927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:00.209999 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.210003 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:47:00.210086 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:47:00.210125 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":59,"thread_start_us":103,"threads_started":1}
I20260928 22:47:00.210212 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.207101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57763 (local address 127.0.98.1:33999)
0928 22:47:00.207143 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.207144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.207377 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:47:00.207413 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.207414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.207440 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.207508 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:00.207658 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.208413 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.209352 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.209848 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.209916 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.209987 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.210095 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.210141 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:47:00.210307 14866 leader_election.cc:302] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e80fa9a88c4830a08b7b0acc2f1bea, d73814f655b44a8aaa7b68cdb6097e06; no voters:
I20260928 22:47:00.210433 15124 raft_consensus.cc:2781] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:47:00.210490 15124 raft_consensus.cc:482] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:47:00.210520 15124 raft_consensus.cc:3037] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:00.210582 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44d3fac3b54447e8e66b7fd3ad32046" candidate_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa316c0af4c341d3be65e785aaab66a6" is_pre_election: true
I20260928 22:47:00.210719 14794 raft_consensus.cc:2454] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e80fa9a88c4830a08b7b0acc2f1bea in term 0.
I20260928 22:47:00.211287 15124 raft_consensus.cc:504] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.211406 15124 leader_election.cc:288] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [CANDIDATE]: Term 1 election: Requested vote from peers d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373), fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999)
I20260928 22:47:00.211547 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44d3fac3b54447e8e66b7fd3ad32046" candidate_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa316c0af4c341d3be65e785aaab66a6"
I20260928 22:47:00.211596 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44d3fac3b54447e8e66b7fd3ad32046" candidate_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73814f655b44a8aaa7b68cdb6097e06"
I20260928 22:47:00.211621 14794 raft_consensus.cc:3037] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:00.211690 15064 raft_consensus.cc:3037] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:00.212363 14794 raft_consensus.cc:2454] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e80fa9a88c4830a08b7b0acc2f1bea in term 1.
I20260928 22:47:00.212549 14868 leader_election.cc:302] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e80fa9a88c4830a08b7b0acc2f1bea, fa316c0af4c341d3be65e785aaab66a6; no voters:
I20260928 22:47:00.212582 15064 raft_consensus.cc:2454] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e80fa9a88c4830a08b7b0acc2f1bea in term 1.
I20260928 22:47:00.212661 15124 raft_consensus.cc:2781] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:47:00.212869 15124 raft_consensus.cc:686] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 LEADER]: Becoming Leader. State: Replica: 92e80fa9a88c4830a08b7b0acc2f1bea, State: Running, Role: LEADER
I20260928 22:47:00.212997 15124 consensus_queue.cc:237] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } }
I20260928 22:47:00.213804 14652 catalog_manager.cc:5495] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea reported cstate change: term changed from 0 to 1, leader changed from <none> to 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2). New cstate: current_term: 1 leader_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e80fa9a88c4830a08b7b0acc2f1bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } }
I20260928 22:47:00.239473 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.237418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.98.2:36261)
0928 22:47:00.237464 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:00.237466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:00.237500 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:47:00.237546 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:00.237546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:00.237570 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:00.237638 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:47:00.237724 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.238184 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.238847 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:00.239034 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:00.239081 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:00.239183 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:47:00.239284 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:47:00.239341 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:00.239354 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 22:47:00.239371 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:47:00.241154 14794 raft_consensus.cc:1261] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Refusing update from remote peer 92e80fa9a88c4830a08b7b0acc2f1bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:00.241144 15064 raft_consensus.cc:1261] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Refusing update from remote peer 92e80fa9a88c4830a08b7b0acc2f1bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:00.241492 15127 consensus_queue.cc:1035] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa316c0af4c341d3be65e785aaab66a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:00.241627 15124 consensus_queue.cc:1035] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73814f655b44a8aaa7b68cdb6097e06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:00.243506 15130 mvcc.cc:205] Tried to move back new op lower bound from 7334443500505169920 to 7334443500393111552. Current Snapshot: MvccSnapshot[applied={T|T < 7334443500505169920}]
I20260928 22:47:00.243829 15131 mvcc.cc:205] Tried to move back new op lower bound from 7334443500505169920 to 7334443500393111552. Current Snapshot: MvccSnapshot[applied={T|T < 7334443500505169920}]
I20260928 22:47:00.245946 14651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:00.246031 14651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:00.246661 14651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Sending DeleteTablet for 3 replicas of tablet e44d3fac3b54447e8e66b7fd3ad32046
I20260928 22:47:00.246915 14774 tablet_service.cc:1514] Processing DeleteTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46336
I20260928 22:47:00.246963 14912 tablet_service.cc:1514] Processing DeleteTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55338
I20260928 22:47:00.247005 14849 tablet_replica.cc:330] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: stopping tablet replica
I20260928 22:47:00.246963 15044 tablet_service.cc:1514] Processing DeleteTablet for tablet e44d3fac3b54447e8e66b7fd3ad32046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260928 22:47:00.247051 14849 raft_consensus.cc:2229] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:00.247103 14849 raft_consensus.cc:2258] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:00.247200 15140 tablet_replica.cc:330] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: stopping tablet replica
I20260928 22:47:00.247288 15140 raft_consensus.cc:2229] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:47:00.247387 15140 raft_consensus.cc:2258] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:00.247546 14849 ts_tablet_manager.cc:1905] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:00.247778 15140 ts_tablet_manager.cc:1905] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:00.248858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14690
I20260928 22:47:00.249476 15139 tablet_replica.cc:330] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: stopping tablet replica
I20260928 22:47:00.249621 15139 raft_consensus.cc:2229] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:00.249807 15139 raft_consensus.cc:2258] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:00.249888 15140 ts_tablet_manager.cc:1918] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:00.249963 15140 log.cc:1196] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/e44d3fac3b54447e8e66b7fd3ad32046
I20260928 22:47:00.250206 15140 ts_tablet_manager.cc:1939] T e44d3fac3b54447e8e66b7fd3ad32046 P 92e80fa9a88c4830a08b7b0acc2f1bea: Deleting consensus metadata
I20260928 22:47:00.250593 14637 catalog_manager.cc:4841] TS 92e80fa9a88c4830a08b7b0acc2f1bea (127.0.98.2:36261): tablet e44d3fac3b54447e8e66b7fd3ad32046 (table test-table [id=28e33e0294504039967b5430ae3478cf]) successfully deleted
I20260928 22:47:00.253365 15139 ts_tablet_manager.cc:1905] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:47:00.253648 14639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33999 (error 108)
I20260928 22:47:00.254635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14845
I20260928 22:47:00.255460 15139 ts_tablet_manager.cc:1918] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:00.255510 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.254734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:46346)
0928 22:47:00.255426 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.255475 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668}
I20260928 22:47:00.255527 15139 log.cc:1196] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/e44d3fac3b54447e8e66b7fd3ad32046
I20260928 22:47:00.255791 15139 ts_tablet_manager.cc:1939] T e44d3fac3b54447e8e66b7fd3ad32046 P d73814f655b44a8aaa7b68cdb6097e06: Deleting consensus metadata
W20260928 22:47:00.255918 14639 catalog_manager.cc:4584] TS fa316c0af4c341d3be65e785aaab66a6 (127.0.98.1:33999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e44d3fac3b54447e8e66b7fd3ad32046: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
I20260928 22:47:00.256565 14637 catalog_manager.cc:4841] TS d73814f655b44a8aaa7b68cdb6097e06 (127.0.98.3:37373): tablet e44d3fac3b54447e8e66b7fd3ad32046 (table test-table [id=28e33e0294504039967b5430ae3478cf]) successfully deleted
I20260928 22:47:00.259555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14981
I20260928 22:47:00.263904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 14619
I20260928 22:47:00.268503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--webserver_interface=127.0.98.62
--webserver_port=34525
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:00.349119 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:00.349365 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:00.349401 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:00.350641 15142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:47:00.350696 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:00.350714 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:00.350730 15142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:47:00.350744 15142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:47:00.352311 15142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:00.352576 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:00.352833 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:00.357522 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.357543 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.357534 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.357590 15142 server_base.cc:1034] running on GCE node
I20260928 22:47:00.357928 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:00.358218 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:00.359365 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790635620359350 us; error 31 us; skew 500 ppm
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:00.361901 15142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:47:00.362340 15142 webserver.cc:460] Webserver started at http://127.0.98.62:34525/ using document root <none> and password file <none>
I20260928 22:47:00.362550 15142 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:00.362600 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:00.363970 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:00.364650 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.364877 15142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:00.364972 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "0dfff2f701894eec8929d77af854a7b3"
format_stamp: "Formatted at 2026-09-28 22:46:58 on dist-test-slave-pxr1"
server_key: "d697d856b6f7f92872b764bdc7ac49c5"
server_key_iv: "915026a11b0220c4973c78348277aa13"
server_key_version: "encryptionkey@0"
I20260928 22:47:00.365355 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:00.381680 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:00.383888 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:00.384016 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:00.387681 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44945
I20260928 22:47:00.387807 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44945 every 8 connection(s)
I20260928 22:47:00.388088 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:47:00.389655 15210 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:47:00.390259 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap starting.
I20260928 22:47:00.393234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15142
I20260928 22:47:00.393519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33999
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=42189
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:00.398988 15210 log.cc:824] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:00.401329 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:47:00.401610 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3: Bootstrap complete.
I20260928 22:47:00.403753 15210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:47:00.404000 15210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Initialized, Role: FOLLOWER
I20260928 22:47:00.404160 15210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:47:00.404237 15210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:47:00.404286 15210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:47:00.404335 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:47:00.405431 15210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:47:00.405539 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dfff2f701894eec8929d77af854a7b3; no voters:
I20260928 22:47:00.405706 15210 leader_election.cc:288] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:47:00.405755 15214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:47:00.405871 15214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [term 3 LEADER]: Becoming Leader. State: Replica: 0dfff2f701894eec8929d77af854a7b3, State: Running, Role: LEADER
I20260928 22:47:00.405967 15214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } }
I20260928 22:47:00.405995 15210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:47:00.406246 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:47:00.406342 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:00.406520 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dfff2f701894eec8929d77af854a7b3. Latest consensus state: current_term: 3 leader_uuid: "0dfff2f701894eec8929d77af854a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dfff2f701894eec8929d77af854a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44945 } } }
I20260928 22:47:00.406574 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dfff2f701894eec8929d77af854a7b3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:00.406566 15219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:47:00.407907 15219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:47:00.408038 15219 catalog_manager.cc:1259] Loaded cluster ID: 5b6e393f716e425f85113b8d4edb03b6
I20260928 22:47:00.408319 15219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 22:47:00.408592 15219 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faeaac0f9ad google::LogMessage::Fail() at ??:0
@ 0x7faeaac1288a google::LogMessage::SendToLog() at ??:0
@ 0x7faeaac0f4de google::LogMessage::Flush() at ??:0
@ 0x7faeaac10a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faeb2b300f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faeb2b398b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faeacadd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faeacad2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faead9366db start_thread at ??:0
@ 0x7faea970d61f clone at ??:0
W20260928 22:47:00.489305 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:00.489518 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:00.489544 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:00.490764 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:00.490808 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:00.490834 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:47:00.492332 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42189
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:00.492570 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:00.492830 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:00.497786 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.498085 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.498220 15212 server_base.cc:1034] running on GCE node
W20260928 22:47:00.497979 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.498528 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:00.498827 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:00.500001 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790635620499984 us; error 37 us; skew 500 ppm
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:00.502897 15212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:47:00.503461 15212 webserver.cc:460] Webserver started at http://127.0.98.1:42189/ using document root <none> and password file <none>
I20260928 22:47:00.503726 15212 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:00.503782 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:00.505213 15212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:47:00.505870 15256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.506037 15212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:47:00.506115 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "fa316c0af4c341d3be65e785aaab66a6"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "91f0229cc1920b58879b7a489f80d033"
server_key_iv: "4a8ae9743939dfe66eb2b53161b4f701"
server_key_version: "encryptionkey@0"
I20260928 22:47:00.506449 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:00.519763 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:00.521843 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:00.521960 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:00.522173 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:00.522575 15263 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 22:47:00.524466 15263 ts_tablet_manager.cc:1852] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 22:47:00.524873 15263 ts_tablet_manager.cc:1905] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:00.527271 15263 ts_tablet_manager.cc:1918] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:00.527365 15263 log.cc:1196] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/e44d3fac3b54447e8e66b7fd3ad32046
I20260928 22:47:00.527634 15263 ts_tablet_manager.cc:1939] T e44d3fac3b54447e8e66b7fd3ad32046 P fa316c0af4c341d3be65e785aaab66a6: Deleting consensus metadata
I20260928 22:47:00.527850 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 22:47:00.527900 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 22:47:00.527941 15212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:00.527966 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.533862 15212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260928 22:47:00.533954 15370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260928 22:47:00.534217 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:47:00.539166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15212
I20260928 22:47:00.539438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36261
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=36491
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:00.606900 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:47:00.534940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:33173)
0928 22:47:00.535123 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.535175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:00.535845 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:00.606681 (+ 70836us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
W20260928 22:47:00.607194 15371 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44945: Client connection negotiation failed: client connection to 127.0.98.62:44945: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:47:00.607534 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.607428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:50299)
0928 22:47:00.607475 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.607513 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 22:47:00.607864 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.607788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:58797)
0928 22:47:00.607827 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.607849 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:47:00.607957 15371 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:47:00.624410 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:00.624619 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:00.624643 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:00.625847 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:00.625895 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:00.625921 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:47:00.627336 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36491
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:00.627539 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:00.627723 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:00.632550 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.632668 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.632560 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.632789 15374 server_base.cc:1034] running on GCE node
I20260928 22:47:00.632956 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:00.633164 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:00.634299 15374 hybrid_clock.cc:648] HybridClock initialized: now 1790635620634286 us; error 27 us; skew 500 ppm
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:00.636588 15374 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:47:00.637077 15374 webserver.cc:460] Webserver started at http://127.0.98.2:36491/ using document root <none> and password file <none>
I20260928 22:47:00.637293 15374 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:00.637341 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:00.638593 15374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:47:00.639083 15389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.639232 15374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:47:00.639310 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "92e80fa9a88c4830a08b7b0acc2f1bea"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a55fdbbbe4262001f0d53cb42d934b65"
server_key_iv: "a09288eac9a2f48f50894a9080bcb2a1"
server_key_version: "encryptionkey@0"
I20260928 22:47:00.639647 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:00.661003 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:00.662773 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:00.662902 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:00.663117 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:00.663393 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:00.663424 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.663453 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:00.663477 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.668965 15374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36261
I20260928 22:47:00.669023 15502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36261 every 8 connection(s)
I20260928 22:47:00.669359 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:47:00.670722 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.670174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:39211)
0928 22:47:00.670479 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.670565 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":80,"threads_started":1}
W20260928 22:47:00.671173 15503 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44945: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
I20260928 22:47:00.671481 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.671388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:44501)
0928 22:47:00.671435 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.671460 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:47:00.671749 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.671674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:35181)
0928 22:47:00.671709 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.671730 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 22:47:00.671844 15503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:47:00.674794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15374
I20260928 22:47:00.675086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37373
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44945
--webserver_port=43985
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:00.759500 15506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:00.759716 15506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:00.759748 15506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:00.760984 15506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:00.761029 15506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:00.761066 15506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:47:00.762579 15506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43985
--tserver_master_addrs=127.0.98.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:00.762835 15506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:00.763051 15506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:00.767915 15514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.767879 15512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:00.767889 15511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:00.768158 15506 server_base.cc:1034] running on GCE node
I20260928 22:47:00.768362 15506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:00.768590 15506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:00.769740 15506 hybrid_clock.cc:648] HybridClock initialized: now 1790635620769722 us; error 33 us; skew 500 ppm
Sep 28 22:47:00 dist-test-slave-pxr1 krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:00.772192 15506 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:47:00.772651 15506 webserver.cc:460] Webserver started at http://127.0.98.3:43985/ using document root <none> and password file <none>
I20260928 22:47:00.772897 15506 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:00.772943 15506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:00.774231 15506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:00.774776 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.774923 15506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:47:00.775010 15506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "d73814f655b44a8aaa7b68cdb6097e06"
format_stamp: "Formatted at 2026-09-28 22:46:59 on dist-test-slave-pxr1"
server_key: "a50783d47090f644776b9a6d5df730ff"
server_key_iv: "40b0464eb07b9f7742a4ccf485fcbced"
server_key_version: "encryptionkey@0"
I20260928 22:47:00.775352 15506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:00.784802 15506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:00.786909 15506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:00.787021 15506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:00.787201 15506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:00.787451 15506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:00.787478 15506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.787499 15506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:00.787510 15506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:00.792995 15506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37373
I20260928 22:47:00.793066 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37373 every 8 connection(s)
I20260928 22:47:00.793375 15506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:47:00.794413 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.794035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:60203)
0928 22:47:00.794195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.794274 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260928 22:47:00.794842 15635 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44945: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
I20260928 22:47:00.795110 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.795026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:34655)
0928 22:47:00.795065 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.795092 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 22:47:00.795359 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:00.795293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:40507)
0928 22:47:00.795325 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:00.795345 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 22:47:00.795459 15635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:47:00.800844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15506
I20260928 22:47:01.608487 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:01.608254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:40475)
0928 22:47:01.608392 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:01.608430 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260928 22:47:01.672319 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:01.672115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:59589)
0928 22:47:01.672226 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:01.672263 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260928 22:47:01.795994 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:01.795786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:33111)
0928 22:47:01.795905 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:01.795945 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 22:47:02.609122 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:02.608914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:44259)
0928 22:47:02.609037 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:02.609073 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260928 22:47:02.672992 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:02.672782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:43947)
0928 22:47:02.672910 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:02.672950 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 22:47:02.796620 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:02.796364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:33331)
0928 22:47:02.796519 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:02.796575 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 22:47:03.609735 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:03.609513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:37861)
0928 22:47:03.609648 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:03.609686 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 22:47:03.673673 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:03.673428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:33267)
0928 22:47:03.673585 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:03.673626 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260928 22:47:03.797292 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:03.797067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:46255)
0928 22:47:03.797207 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:03.797241 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260928 22:47:04.610399 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:04.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:48471)
0928 22:47:04.610311 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:04.610351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 22:47:04.674346 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:04.674144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:52393)
0928 22:47:04.674271 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:04.674307 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 22:47:04.797925 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:04.797698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:35697)
0928 22:47:04.797845 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:04.797885 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260928 22:47:05.611003 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:05.610809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:41839)
0928 22:47:05.610924 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:05.610958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260928 22:47:05.674968 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:05.674782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:35883)
0928 22:47:05.674889 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:05.674923 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260928 22:47:05.798580 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:05.798356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:50325)
0928 22:47:05.798495 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:05.798532 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 22:47:06.611603 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:06.611383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:43361)
0928 22:47:06.611519 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:06.611556 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260928 22:47:06.675544 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:06.675352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:57529)
0928 22:47:06.675461 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:06.675498 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260928 22:47:06.799201 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:06.799003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:48297)
0928 22:47:06.799127 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:06.799159 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260928 22:47:07.612221 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:07.612011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:37385)
0928 22:47:07.612126 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:07.612163 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 22:47:07.676079 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:07.675890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:53775)
0928 22:47:07.675998 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:07.676035 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260928 22:47:07.799845 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:07.799637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:50663)
0928 22:47:07.799752 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:07.799793 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 22:47:08.612888 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:08.612595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:37003)
0928 22:47:08.612793 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:08.612840 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260928 22:47:08.676683 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:08.676470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:58801)
0928 22:47:08.676592 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:08.676632 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260928 22:47:08.800607 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:08.800351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:45341)
0928 22:47:08.800518 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:08.800560 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260928 22:47:09.613670 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:09.613400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:33435)
0928 22:47:09.613549 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:09.613612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260928 22:47:09.677289 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:09.677083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:35371)
0928 22:47:09.677203 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:09.677239 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 22:47:09.801294 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:09.801086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:52423)
0928 22:47:09.801198 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:09.801242 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 22:47:10.614354 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:10.614125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:41815)
0928 22:47:10.614261 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:10.614306 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 22:47:10.677891 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:10.677698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:45449)
0928 22:47:10.677808 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:10.677842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260928 22:47:10.802016 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:10.801751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:34373)
0928 22:47:10.801900 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:10.801965 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260928 22:47:11.614948 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:11.614730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:43515)
0928 22:47:11.614854 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:11.614899 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260928 22:47:11.678603 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:11.678372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:45395)
0928 22:47:11.678511 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:11.678554 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260928 22:47:11.802683 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:11.802467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:33039)
0928 22:47:11.802599 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:11.802635 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260928 22:47:12.615657 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:12.615406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:52319)
0928 22:47:12.615565 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:12.615610 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260928 22:47:12.679224 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:12.679023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:54971)
0928 22:47:12.679134 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:12.679171 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260928 22:47:12.803397 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:12.803210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:33379)
0928 22:47:12.803323 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:12.803349 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 22:47:13.616475 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:13.616222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:55957)
0928 22:47:13.616384 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:13.616427 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 22:47:13.679865 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:13.679670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:54401)
0928 22:47:13.679779 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:13.679813 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260928 22:47:13.804189 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:13.803908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:34783)
0928 22:47:13.804085 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:13.804132 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260928 22:47:14.617230 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:14.616977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:58669)
0928 22:47:14.617129 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:14.617181 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260928 22:47:14.680583 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:14.680358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:36801)
0928 22:47:14.680487 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:14.680529 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 22:47:14.805001 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:14.804788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:38325)
0928 22:47:14.804908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:14.804953 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 22:47:15.617998 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:15.617782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.1:33741)
0928 22:47:15.617918 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:15.617951 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260928 22:47:15.681317 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:15.681096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.2:47847)
0928 22:47:15.681233 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:15.681269 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260928 22:47:15.805771 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:15.805510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44945 (local address 127.0.98.3:43969)
0928 22:47:15.805667 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:15.805713 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260928 22:47:15.806746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15212
I20260928 22:47:15.810748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15374
I20260928 22:47:15.814838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15506
2026-09-28T22:47:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18008 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:47:15 dist-test-slave-pxr1 krb5kdc[15686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:47:15 dist-test-slave-pxr1 krb5kdc[15686](info): set up 2 sockets
Sep 28 22:47:15 dist-test-slave-pxr1 krb5kdc[15686](info): commencing operation
krb5kdc: starting...
W20260928 22:47:16.780617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:47:16 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:47:16.899151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:16.986203 15699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:16.986416 15699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:16.986438 15699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:16.987609 15699 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:47:16.987655 15699 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:47:16.987666 15699 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:47:16.987677 15699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:47:16.987689 15699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:16.987699 15699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:16.987709 15699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:47:16.987721 15699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:47:16.989271 15699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:16.989522 15699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:16.989738 15699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:16.994997 15704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:16.995064 15705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:16.995090 15699 server_base.cc:1034] running on GCE node
W20260928 22:47:16.995064 15707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:16.995452 15699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:16.995785 15699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:16.996923 15699 hybrid_clock.cc:648] HybridClock initialized: now 1790635636996909 us; error 33 us; skew 500 ppm
Sep 28 22:47:16 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:16.999333 15699 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:47:16.999791 15699 webserver.cc:460] Webserver started at http://127.0.98.62:34277/ using document root <none> and password file <none>
I20260928 22:47:16.999989 15699 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:17.000038 15699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:17.000167 15699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:47:17.001259 15699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "7aace59e680e454bb30006a99890f577"
format_stamp: "Formatted at 2026-09-28 22:47:16 on dist-test-slave-pxr1"
server_key: "871100bb6e030b6bd0e6563839a03d4a"
server_key_iv: "456a638e3b95cc25db81199bfb809d16"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.001597 15699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aace59e680e454bb30006a99890f577"
format_stamp: "Formatted at 2026-09-28 22:47:16 on dist-test-slave-pxr1"
server_key: "871100bb6e030b6bd0e6563839a03d4a"
server_key_iv: "456a638e3b95cc25db81199bfb809d16"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.002908 15699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:47:17.003727 15714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.003948 15699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:17.004053 15699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7aace59e680e454bb30006a99890f577"
format_stamp: "Formatted at 2026-09-28 22:47:16 on dist-test-slave-pxr1"
server_key: "871100bb6e030b6bd0e6563839a03d4a"
server_key_iv: "456a638e3b95cc25db81199bfb809d16"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.004117 15699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:17.014901 15699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:17.017089 15699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:17.017231 15699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:17.020980 15699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33439
I20260928 22:47:17.021018 15766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33439 every 8 connection(s)
I20260928 22:47:17.021342 15699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:47:17.022137 15767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:17.023655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15699
I20260928 22:47:17.023764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:47:17.023957 392 external_mini_cluster.cc:1428] Setting key ad3b2a9144292141facc7c12138a1760
I20260928 22:47:17.024616 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap starting.
I20260928 22:47:17.025204 15767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:17.025516 15767 log.cc:824] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:17.026221 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: No bootstrap required, opened a new log
I20260928 22:47:17.028249 15767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:17.028391 15767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:17.028422 15767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aace59e680e454bb30006a99890f577, State: Initialized, Role: FOLLOWER
I20260928 22:47:17.028570 15767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:17.028643 15767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:47:17.028673 15767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:47:17.028743 15767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:17.029457 15767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:17.029580 15767 leader_election.cc:302] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aace59e680e454bb30006a99890f577; no voters:
I20260928 22:47:17.029762 15767 leader_election.cc:288] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:47:17.029806 15772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:47:17.029968 15772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 LEADER]: Becoming Leader. State: Replica: 7aace59e680e454bb30006a99890f577, State: Running, Role: LEADER
I20260928 22:47:17.030054 15767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:47:17.030061 15772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:17.030422 15772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aace59e680e454bb30006a99890f577. Latest consensus state: current_term: 1 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:17.030490 15772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:17.030725 15773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:17.030795 15773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:17.030915 15778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:47:17.031399 15778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:47:17.033056 15778 catalog_manager.cc:1347] Generated new cluster ID: 5b5adc09fc804609aeef80545e21c2a6
I20260928 22:47:17.033109 15778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:47:17.034585 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.024449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.98.62:33439)
0928 22:47:17.024614 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.024623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.024649 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:47:17.025109 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.025116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.025163 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.025277 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:17.026026 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.026943 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.027988 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.028383 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.028476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.031257 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:17.031262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:17.031262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:17.031275 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:17.033182 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.033498 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.033501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.033501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.033604 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.033774 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.033775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.033775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.034013 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:17.034136 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.034238 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.034368 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260928 22:47:17.053583 15778 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:47:17.054162 15778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:47:17.057694 15778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Generated new TSK 0
I20260928 22:47:17.057900 15778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:47:17.091497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:17.173298 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:17.173534 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:17.173568 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:17.174753 15793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:17.174795 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:17.174830 15793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:47:17.176306 15793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:17.176545 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:17.176781 15793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:17.181622 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:17.181759 15793 server_base.cc:1034] running on GCE node
W20260928 22:47:17.181622 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:17.181675 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:17.182099 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:17.182354 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:17.183532 15793 hybrid_clock.cc:648] HybridClock initialized: now 1790635637183512 us; error 34 us; skew 500 ppm
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:17.186058 15793 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:47:17.186472 15793 webserver.cc:460] Webserver started at http://127.0.98.1:39591/ using document root <none> and password file <none>
I20260928 22:47:17.186666 15793 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:17.186709 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:17.186830 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:47:17.187870 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f1f28c3c4534606b53ac0d66c300b24"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "28343f55d19dfd2c74ec534f899587b8"
server_key_iv: "1bf46d7faca3f38dec91374114e34998"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.188205 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1f28c3c4534606b53ac0d66c300b24"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "28343f55d19dfd2c74ec534f899587b8"
server_key_iv: "1bf46d7faca3f38dec91374114e34998"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.189566 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:47:17.190307 15808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.190472 15793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:47:17.190548 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "8f1f28c3c4534606b53ac0d66c300b24"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "28343f55d19dfd2c74ec534f899587b8"
server_key_iv: "1bf46d7faca3f38dec91374114e34998"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.190618 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:17.200945 15793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:17.203073 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:17.203192 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:17.203413 15793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:17.203704 15793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:17.203737 15793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.203766 15793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:17.203789 15793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.209338 15793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34141
I20260928 22:47:17.209379 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34141 every 8 connection(s)
I20260928 22:47:17.209686 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:17.216687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15793
I20260928 22:47:17.216976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:47:17.217211 392 external_mini_cluster.cc:1428] Setting key 021e157ffbb7d7065ec67965a3bfad92
I20260928 22:47:17.219784 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.210404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52529 (local address 127.0.98.62:33439)
0928 22:47:17.210468 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.210470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.210655 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:47:17.211128 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.211129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.211139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.211195 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:17.212161 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.212694 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.213818 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.214043 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.214110 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.216451 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:17.216453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:17.216454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:17.216463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:17.218206 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.218635 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.218636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.218637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.218729 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.219062 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.219063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.219063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.219200 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:17.219271 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.219639 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.219715 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:47:17.220045 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.210352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:52529)
0928 22:47:17.210502 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.210540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.211018 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.211251 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.211254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.211833 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:17.212039 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.212045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.212826 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.212827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.213701 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.213704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.214152 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.214153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.214273 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.214832 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:17.214841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:17.216333 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:17.218303 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.218312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.218314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.218562 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.218872 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.218873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.218874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.218998 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.219297 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:17.219304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:17.219503 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.219623 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.219813 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 22:47:17.220634 15922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:17.220780 15922 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:17.220983 15922 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:17.221557 15731 ts_manager.cc:194] Registered new tserver with Master: 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141)
I20260928 22:47:17.222164 15731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52529
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:47:17.242774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:17.324278 15927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:17.324502 15927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:17.324527 15927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:17.325729 15927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:17.325779 15927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:17.325807 15927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:47:17.327291 15927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:17.327524 15927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:17.327725 15927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:47:17.332551 15927 server_base.cc:1034] running on GCE node
W20260928 22:47:17.332526 15932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:17.332683 15933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:17.332609 15935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:17.332957 15927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:17.333187 15927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:17.334354 15927 hybrid_clock.cc:648] HybridClock initialized: now 1790635637334332 us; error 34 us; skew 500 ppm
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:17.336766 15927 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:47:17.337249 15927 webserver.cc:460] Webserver started at http://127.0.98.2:36345/ using document root <none> and password file <none>
I20260928 22:47:17.337471 15927 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:17.337518 15927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:17.337646 15927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:47:17.338671 15927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "db31495757d74840bb382f49e9d13c0a"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "9f60187207dcd2a186e0bc47fc054ce6"
server_key_iv: "eba294e939a62a9b1682fa67f41dfe18"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.339001 15927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db31495757d74840bb382f49e9d13c0a"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "9f60187207dcd2a186e0bc47fc054ce6"
server_key_iv: "eba294e939a62a9b1682fa67f41dfe18"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.340284 15927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:47:17.341038 15942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.341173 15927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:47:17.341259 15927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "db31495757d74840bb382f49e9d13c0a"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "9f60187207dcd2a186e0bc47fc054ce6"
server_key_iv: "eba294e939a62a9b1682fa67f41dfe18"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.341324 15927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:17.359509 15927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:17.361763 15927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:17.361907 15927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:17.362128 15927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:17.362438 15927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:17.362470 15927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.362498 15927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:17.362524 15927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.368137 15927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41583
I20260928 22:47:17.368184 16055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41583 every 8 connection(s)
I20260928 22:47:17.368496 15927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:17.376931 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.369166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53775 (local address 127.0.98.62:33439)
0928 22:47:17.369226 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.369228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.369428 (+ 200us) server_negotiation.cc:407] Connection header received
0928 22:47:17.370369 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.370370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.370382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.370436 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:17.371045 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.371550 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.372477 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.372672 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.372768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.374767 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:17.374769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:17.374769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:17.374776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:17.375832 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.376182 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.376183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.376183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.376228 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.376408 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.376409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.376409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.376503 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:17.376544 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.376798 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.376861 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:47:17.377110 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.369147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:53775)
0928 22:47:17.369307 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.369343 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.370262 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.370501 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.370504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.370748 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:17.370961 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.370965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.371613 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.371614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.372405 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.372408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.372759 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.372760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.372885 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.373345 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:17.373353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:17.374675 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:17.375897 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.375902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.375905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.376126 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.376272 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.376272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.376272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.376363 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.376543 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:17.376546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:17.376660 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.376784 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.376910 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 22:47:17.377561 16056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:17.377650 16056 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:17.377664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 15927
I20260928 22:47:17.377764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:47:17.377830 16056 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:17.377956 392 external_mini_cluster.cc:1428] Setting key b54a32582df6f88bacca966dd62f66cc
I20260928 22:47:17.378203 15731 ts_manager.cc:194] Registered new tserver with Master: db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
I20260928 22:47:17.378711 15731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53775
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:47:17.403136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:17.484380 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:17.484612 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:17.484638 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:17.485917 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:17.485975 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:17.486009 16061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:47:17.487546 16061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:17.487751 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:17.487941 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:17.492815 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:17.493034 16061 server_base.cc:1034] running on GCE node
W20260928 22:47:17.492885 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:17.492782 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:17.493340 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:17.493593 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:17.494753 16061 hybrid_clock.cc:648] HybridClock initialized: now 1790635637494739 us; error 32 us; skew 500 ppm
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:17.497327 16061 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:47:17.497798 16061 webserver.cc:460] Webserver started at http://127.0.98.3:40569/ using document root <none> and password file <none>
I20260928 22:47:17.498021 16061 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:17.498070 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:17.498188 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:47:17.499260 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2bace48c44a46d6b773f3de81c73f3d"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "7b7a9d0c5b4719c26e3bf7934e8c46ad"
server_key_iv: "4651a40b0f468a83425736b24455f067"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.499591 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2bace48c44a46d6b773f3de81c73f3d"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "7b7a9d0c5b4719c26e3bf7934e8c46ad"
server_key_iv: "4651a40b0f468a83425736b24455f067"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.500926 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:47:17.501639 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.501791 16061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.501868 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "b2bace48c44a46d6b773f3de81c73f3d"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "7b7a9d0c5b4719c26e3bf7934e8c46ad"
server_key_iv: "4651a40b0f468a83425736b24455f067"
server_key_version: "encryptionkey@0"
I20260928 22:47:17.501931 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:17.519914 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:17.522071 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:17.522217 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:17.522439 16061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:17.522749 16061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:17.522781 16061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.522811 16061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:17.522833 16061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:17.529716 16061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45845
I20260928 22:47:17.529824 16189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45845 every 8 connection(s)
I20260928 22:47:17.530113 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:47:17 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:17.538067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16061
I20260928 22:47:17.538172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:47:17.538342 392 external_mini_cluster.cc:1428] Setting key 5150b726716d33e84411ddb964a66c87
I20260928 22:47:17.539968 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.531187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37641 (local address 127.0.98.62:33439)
0928 22:47:17.531240 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.531242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.531501 (+ 259us) server_negotiation.cc:407] Connection header received
0928 22:47:17.532062 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.532063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.532075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.532123 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:17.533364 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.534025 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.535035 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.535217 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.535269 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.537339 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:17.537340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:17.537341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:17.537348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:17.538700 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.539102 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.539103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.539103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.539162 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.539376 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.539377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.539377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.539493 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:17.539548 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.539804 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.539881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:47:17.540154 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.531221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:37641)
0928 22:47:17.531379 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.531427 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.531961 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.532194 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.532213 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.532973 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:17.533266 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.533274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.534138 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.534139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.534960 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.534963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.535278 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.535278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.535374 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.535881 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:17.535890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:17.537230 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:17.538764 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.538771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.538774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.539034 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.539216 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:17.539217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:17.539217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:17.539323 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:17.539546 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:17.539552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:17.539686 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.539775 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.539931 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20260928 22:47:17.540547 16190 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:17.540632 16190 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:17.540829 16190 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:17.541180 15731 ts_manager.cc:194] Registered new tserver with Master: b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845)
I20260928 22:47:17.541545 15731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37641
I20260928 22:47:17.550112 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:47:17.557011 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.551415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.62:33439)
0928 22:47:17.551463 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.551465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.551519 (+ 54us) server_negotiation.cc:407] Connection header received
0928 22:47:17.551576 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.551577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.551585 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.551628 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:17.552198 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.552689 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.553430 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.553626 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.553680 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.554792 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:17.554793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:17.554794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:17.554802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:17.556020 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.556323 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.556324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.556325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.556376 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:17.556513 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:17.556513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:17.556513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:17.556654 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:17.556757 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.556866 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.556928 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:47:17.558488 15731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:47:17.560042 15731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:47:17.566428 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.98.1:34141)
0928 22:47:17.562673 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.562678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.562751 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:47:17.562839 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.562842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.562858 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.562934 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:17.563156 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.564316 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.565568 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.566035 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.566103 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.566163 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.566302 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.566343 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 22:47:17.566457 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34141 (local address 127.0.0.1:34788)
0928 22:47:17.562469 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.562706 (+ 237us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.562758 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.562958 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.562961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.562963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:17.563036 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.563039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.564424 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.564425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.565502 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.565504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.566116 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.566117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.566205 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.566219 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.566282 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.566338 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 22:47:17.566469 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.0.1:44238)
0928 22:47:17.562501 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.562526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.562591 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.563063 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.563064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.563066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:17.563108 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.563110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.564038 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.564039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.565250 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.565252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.566201 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.566202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.566274 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.566278 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.566318 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.566361 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260928 22:47:17.566685 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.0.98.2:41583)
0928 22:47:17.562829 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.562833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.562841 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:47:17.562895 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.562897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.562911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.562991 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:17.563178 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.563945 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.565344 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.566057 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.566121 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.566170 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.566567 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.566606 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 22:47:17.567384 15856 tablet_service.cc:1467] Processing CreateTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 (DEFAULT_TABLE table=test-table [id=52351d481e814c7a8dd4b50c26a98b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:17.567380 15990 tablet_service.cc:1467] Processing CreateTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 (DEFAULT_TABLE table=test-table [id=52351d481e814c7a8dd4b50c26a98b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:17.567593 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45845 (local address 127.0.0.1:48120)
0928 22:47:17.562677 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.562683 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.562719 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.563100 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.563101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.563103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:17.563164 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.563165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.565586 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.565587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.566612 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.566615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.567356 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.567357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.567432 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.567436 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.567489 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.567526 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":102,"threads_started":1}
I20260928 22:47:17.567610 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.562562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.98.3:45845)
0928 22:47:17.562866 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.562870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.562881 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:47:17.562929 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.562931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.562944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.563024 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:17.563241 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.564321 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.566698 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.567281 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.567342 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.567392 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.567501 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.567536 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":85,"threads_started":1}
I20260928 22:47:17.567730 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b8d4a97ba4411fbd49c28d19beddf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:17.567731 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b8d4a97ba4411fbd49c28d19beddf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:17.569203 16124 tablet_service.cc:1467] Processing CreateTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 (DEFAULT_TABLE table=test-table [id=52351d481e814c7a8dd4b50c26a98b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:17.569447 16124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b8d4a97ba4411fbd49c28d19beddf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:17.570995 16207 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap starting.
I20260928 22:47:17.571631 16207 tablet_bootstrap.cc:654] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:17.571942 16209 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Bootstrap starting.
I20260928 22:47:17.572155 16207 log.cc:824] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:17.572446 16208 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Bootstrap starting.
I20260928 22:47:17.572705 16209 tablet_bootstrap.cc:654] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:17.573042 16207 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: No bootstrap required, opened a new log
I20260928 22:47:17.573040 16208 tablet_bootstrap.cc:654] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:17.573109 16207 ts_tablet_manager.cc:1397] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:47:17.573262 16209 log.cc:824] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:17.573577 16208 log.cc:824] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:17.573930 16209 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: No bootstrap required, opened a new log
I20260928 22:47:17.574000 16209 ts_tablet_manager.cc:1397] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:47:17.574625 16208 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: No bootstrap required, opened a new log
I20260928 22:47:17.574694 16208 ts_tablet_manager.cc:1397] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:47:17.574918 16207 raft_consensus.cc:348] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.575094 16207 raft_consensus.cc:374] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:17.575126 16207 raft_consensus.cc:729] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1f28c3c4534606b53ac0d66c300b24, State: Initialized, Role: FOLLOWER
I20260928 22:47:17.575263 16207 consensus_queue.cc:260] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.575555 15922 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
I20260928 22:47:17.575541 16207 ts_tablet_manager.cc:1428] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:47:17.575634 16209 raft_consensus.cc:348] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.575775 16209 raft_consensus.cc:374] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:17.576007 16209 raft_consensus.cc:729] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2bace48c44a46d6b773f3de81c73f3d, State: Initialized, Role: FOLLOWER
I20260928 22:47:17.576153 16209 consensus_queue.cc:260] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.576444 16209 ts_tablet_manager.cc:1428] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:47:17.576526 16190 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
I20260928 22:47:17.576558 16208 raft_consensus.cc:348] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.576710 16208 raft_consensus.cc:374] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:17.576771 16208 raft_consensus.cc:729] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db31495757d74840bb382f49e9d13c0a, State: Initialized, Role: FOLLOWER
I20260928 22:47:17.576913 16208 consensus_queue.cc:260] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.577172 16208 ts_tablet_manager.cc:1428] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:47:17.577191 16056 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
W20260928 22:47:17.619087 16057 tablet.cc:2369] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:47:17.710449 15923 tablet.cc:2369] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:47:17.780833 16191 tablet.cc:2369] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:47:17.913411 16213 raft_consensus.cc:482] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:47:17.913537 16213 raft_consensus.cc:504] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.913868 16213 leader_election.cc:288] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583), b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845)
I20260928 22:47:17.916551 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.914091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38783 (local address 127.0.98.2:41583)
0928 22:47:17.914139 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.914141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.914226 (+ 85us) server_negotiation.cc:407] Connection header received
0928 22:47:17.914283 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.914285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.914297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.914342 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:17.914482 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.914949 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.915894 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.916241 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.916300 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.916310 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.916440 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.916476 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:47:17.916569 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.914040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.98.1:38783)
0928 22:47:17.914159 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.914168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.914229 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.914349 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.914350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.914351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:17.914426 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.914428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.915021 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.915022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.915805 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.915807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.916310 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.916311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.916379 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.916382 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.916428 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.916476 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":16,"thread_start_us":59,"threads_started":1}
I20260928 22:47:17.916617 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.914141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53259 (local address 127.0.98.3:45845)
0928 22:47:17.914185 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:17.914186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:17.914308 (+ 122us) server_negotiation.cc:407] Connection header received
0928 22:47:17.914359 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:17.914360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:17.914372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:17.914422 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:17.914566 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.915018 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.915932 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.916296 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.916357 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.916367 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:17.916515 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:47:17.916555 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:47:17.916671 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:17.914056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45845 (local address 127.0.98.1:53259)
0928 22:47:17.914245 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:17.914253 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:17.914305 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:17.914430 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:17.914430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:17.914432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:17.914505 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.914507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.915090 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.915091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.915846 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:17.915849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:17.916377 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:17.916378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:17.916448 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:17.916455 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:47:17.916499 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:47:17.916538 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":50,"threads_started":1}
I20260928 22:47:17.916986 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b8d4a97ba4411fbd49c28d19beddf4" candidate_uuid: "8f1f28c3c4534606b53ac0d66c300b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2bace48c44a46d6b773f3de81c73f3d" is_pre_election: true
I20260928 22:47:17.917052 16010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b8d4a97ba4411fbd49c28d19beddf4" candidate_uuid: "8f1f28c3c4534606b53ac0d66c300b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db31495757d74840bb382f49e9d13c0a" is_pre_election: true
I20260928 22:47:17.917164 16010 raft_consensus.cc:2454] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1f28c3c4534606b53ac0d66c300b24 in term 0.
I20260928 22:47:17.917158 16144 raft_consensus.cc:2454] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1f28c3c4534606b53ac0d66c300b24 in term 0.
I20260928 22:47:17.917398 15809 leader_election.cc:302] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24, b2bace48c44a46d6b773f3de81c73f3d; no voters:
I20260928 22:47:17.917559 16213 raft_consensus.cc:2781] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:47:17.917614 16213 raft_consensus.cc:482] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:47:17.917635 16213 raft_consensus.cc:3037] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:17.918299 16213 raft_consensus.cc:504] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.918483 16213 leader_election.cc:288] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 1 election: Requested vote from peers db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583), b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845)
I20260928 22:47:17.918550 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b8d4a97ba4411fbd49c28d19beddf4" candidate_uuid: "8f1f28c3c4534606b53ac0d66c300b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2bace48c44a46d6b773f3de81c73f3d"
I20260928 22:47:17.918572 16010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b8d4a97ba4411fbd49c28d19beddf4" candidate_uuid: "8f1f28c3c4534606b53ac0d66c300b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db31495757d74840bb382f49e9d13c0a"
I20260928 22:47:17.918634 16144 raft_consensus.cc:3037] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:17.918643 16010 raft_consensus.cc:3037] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:17.919299 16010 raft_consensus.cc:2454] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1f28c3c4534606b53ac0d66c300b24 in term 1.
I20260928 22:47:17.919307 16144 raft_consensus.cc:2454] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1f28c3c4534606b53ac0d66c300b24 in term 1.
I20260928 22:47:17.919461 15810 leader_election.cc:302] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24, db31495757d74840bb382f49e9d13c0a; no voters:
I20260928 22:47:17.919559 16213 raft_consensus.cc:2781] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:47:17.919711 16213 raft_consensus.cc:686] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1f28c3c4534606b53ac0d66c300b24, State: Running, Role: LEADER
I20260928 22:47:17.919798 16213 consensus_queue.cc:237] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:17.920607 15730 catalog_manager.cc:5495] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8f1f28c3c4534606b53ac0d66c300b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } health_report { overall_health: UNKNOWN } } }
I20260928 22:47:18.009630 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:18.007569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.98.1:34141)
0928 22:47:18.007617 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:18.007619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:18.007629 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:47:18.007671 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:18.007672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:18.007691 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:18.007765 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:47:18.007864 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:18.008325 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:18.009073 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:18.009254 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:18.009296 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:18.009387 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:47:18.009487 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:47:18.009541 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:18.009553 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 22:47:18.009570 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:47:18.011253 16010 raft_consensus.cc:1261] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Refusing update from remote peer 8f1f28c3c4534606b53ac0d66c300b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:18.011255 16144 raft_consensus.cc:1261] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Refusing update from remote peer 8f1f28c3c4534606b53ac0d66c300b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:18.011591 16218 consensus_queue.cc:1035] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:18.011720 16213 consensus_queue.cc:1035] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:18.013965 16221 mvcc.cc:205] Tried to move back new op lower bound from 7334443573291700224 to 7334443572920070144. Current Snapshot: MvccSnapshot[applied={T|T < 7334443573291700224}]
I20260928 22:47:18.014030 16223 mvcc.cc:205] Tried to move back new op lower bound from 7334443573291700224 to 7334443572920070144. Current Snapshot: MvccSnapshot[applied={T|T < 7334443573291700224}]
I20260928 22:47:18.014940 16222 mvcc.cc:205] Tried to move back new op lower bound from 7334443573291700224 to 7334443572920070144. Current Snapshot: MvccSnapshot[applied={T|T < 7334443573291700224 or (T in {7334443573291700224})}]
I20260928 22:47:18.016052 15731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:18.016130 15731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:18.016800 15731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Sending DeleteTablet for 3 replicas of tablet 04b8d4a97ba4411fbd49c28d19beddf4
I20260928 22:47:18.017107 16124 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48120
I20260928 22:47:18.017107 15990 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44238
I20260928 22:47:18.017336 16231 tablet_replica.cc:330] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: stopping tablet replica
I20260928 22:47:18.017438 16231 raft_consensus.cc:2229] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:18.017472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15793
I20260928 22:47:18.017445 15856 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34788
I20260928 22:47:18.017535 16231 raft_consensus.cc:2258] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:18.017680 16232 tablet_replica.cc:330] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: stopping tablet replica
I20260928 22:47:18.017764 16232 raft_consensus.cc:2229] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:18.017859 16232 raft_consensus.cc:2258] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:18.018507 16231 ts_tablet_manager.cc:1905] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:18.019330 16232 ts_tablet_manager.cc:1905] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:18.020182 16231 ts_tablet_manager.cc:1918] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:18.020251 16231 log.cc:1196] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/04b8d4a97ba4411fbd49c28d19beddf4
I20260928 22:47:18.020478 16231 ts_tablet_manager.cc:1939] T 04b8d4a97ba4411fbd49c28d19beddf4 P b2bace48c44a46d6b773f3de81c73f3d: Deleting consensus metadata
I20260928 22:47:18.020918 16232 ts_tablet_manager.cc:1918] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:18.020916 15715 catalog_manager.cc:4841] TS b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): tablet 04b8d4a97ba4411fbd49c28d19beddf4 (table test-table [id=52351d481e814c7a8dd4b50c26a98b21]) successfully deleted
I20260928 22:47:18.021027 16232 log.cc:1196] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/04b8d4a97ba4411fbd49c28d19beddf4
I20260928 22:47:18.021265 16232 ts_tablet_manager.cc:1939] T 04b8d4a97ba4411fbd49c28d19beddf4 P db31495757d74840bb382f49e9d13c0a: Deleting consensus metadata
I20260928 22:47:18.021641 15716 catalog_manager.cc:4841] TS db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): tablet 04b8d4a97ba4411fbd49c28d19beddf4 (table test-table [id=52351d481e814c7a8dd4b50c26a98b21]) successfully deleted
W20260928 22:47:18.023624 15718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34141 (error 108)
I20260928 22:47:18.023883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15927
I20260928 22:47:18.024395 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:18.023940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34141 (local address 127.0.0.1:34802)
0928 22:47:18.024313 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:18.024354 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260928 22:47:18.024509 15718 catalog_manager.cc:4584] TS 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04b8d4a97ba4411fbd49c28d19beddf4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34141: connect: Connection refused (error 111)
I20260928 22:47:18.028457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16061
I20260928 22:47:18.032294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 15699
I20260928 22:47:38.037005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--webserver_interface=127.0.98.62
--webserver_port=34277
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:38.117731 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:38.117951 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:38.117974 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:38.119123 16234 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:47:38.119161 16234 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:47:38.119174 16234 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:47:38.119184 16234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:47:38.119194 16234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:38.119206 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:38.119216 16234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:47:38.119227 16234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:47:38.120687 16234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:38.120922 16234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:38.121119 16234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:38.129308 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.129290 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.129294 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:38.129490 16234 server_base.cc:1034] running on GCE node
I20260928 22:47:38.129734 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:38.130028 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:38.131162 16234 hybrid_clock.cc:648] HybridClock initialized: now 1790635658131155 us; error 29 us; skew 500 ppm
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:38.133682 16234 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:47:38.134158 16234 webserver.cc:460] Webserver started at http://127.0.98.62:34277/ using document root <none> and password file <none>
I20260928 22:47:38.134411 16234 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:38.134472 16234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:38.135810 16234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:47:38.136484 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.136657 16234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:47:38.136771 16234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7aace59e680e454bb30006a99890f577"
format_stamp: "Formatted at 2026-09-28 22:47:16 on dist-test-slave-pxr1"
server_key: "871100bb6e030b6bd0e6563839a03d4a"
server_key_iv: "456a638e3b95cc25db81199bfb809d16"
server_key_version: "encryptionkey@0"
I20260928 22:47:38.137184 16234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:38.161693 16234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:38.163584 16234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:38.163692 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:38.167510 16234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33439
I20260928 22:47:38.167555 16302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33439 every 8 connection(s)
I20260928 22:47:38.167929 16234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:47:38.169430 16303 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:47:38.170001 16303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap starting.
I20260928 22:47:38.172098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16234
I20260928 22:47:38.172348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34141
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=39591
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:38.178431 16303 log.cc:824] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:38.180413 16303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:47:38.180688 16303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap complete.
I20260928 22:47:38.182853 16303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:38.183094 16303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aace59e680e454bb30006a99890f577, State: Initialized, Role: FOLLOWER
I20260928 22:47:38.183238 16303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:38.183305 16303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:47:38.183350 16303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:47:38.183396 16303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:47:38.184422 16303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:38.184512 16303 leader_election.cc:302] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aace59e680e454bb30006a99890f577; no voters:
I20260928 22:47:38.184674 16303 leader_election.cc:288] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:47:38.184751 16307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:47:38.184873 16307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 LEADER]: Becoming Leader. State: Replica: 7aace59e680e454bb30006a99890f577, State: Running, Role: LEADER
I20260928 22:47:38.184960 16303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:47:38.185024 16307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:38.185326 16309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:38.185352 16307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aace59e680e454bb30006a99890f577. Latest consensus state: current_term: 2 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:38.185422 16309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:38.185432 16307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:38.185945 16315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:47:38.186741 16315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:47:38.186846 16315 catalog_manager.cc:1259] Loaded cluster ID: 5b5adc09fc804609aeef80545e21c2a6
I20260928 22:47:38.186921 16315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:47:38.187826 16315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:47:38.187995 16315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Loaded TSK: 0
I20260928 22:47:38.203380 16315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Generated new TSK 1
I20260928 22:47:38.203506 16315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:47:38.268684 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:38.268949 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:38.268983 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:38.270206 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:38.270256 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:38.270293 16305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:47:38.272022 16305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39591
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:38.272306 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:38.272531 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:38.277444 16331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.277444 16333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.277592 16330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:38.277850 16305 server_base.cc:1034] running on GCE node
I20260928 22:47:38.278122 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:38.278355 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:38.279498 16305 hybrid_clock.cc:648] HybridClock initialized: now 1790635658279483 us; error 33 us; skew 500 ppm
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:38.282147 16305 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:47:38.282691 16305 webserver.cc:460] Webserver started at http://127.0.98.1:39591/ using document root <none> and password file <none>
I20260928 22:47:38.282968 16305 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:38.283052 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:38.284787 16305 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:47:38.285600 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.285765 16305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:38.285862 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "8f1f28c3c4534606b53ac0d66c300b24"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "28343f55d19dfd2c74ec534f899587b8"
server_key_iv: "1bf46d7faca3f38dec91374114e34998"
server_key_version: "encryptionkey@0"
I20260928 22:47:38.286307 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:38.317457 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:38.323221 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:38.323410 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:38.323669 16305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:38.324115 16347 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:47:38.325601 16305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:47:38.325659 16305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:47:38.325745 16305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:47:38.326653 16305 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:47:38.326697 16305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:47:38.326756 16347 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap starting.
I20260928 22:47:38.333402 16305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34141
I20260928 22:47:38.333660 16454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34141 every 8 connection(s)
I20260928 22:47:38.333941 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:47:38.336874 16347 log.cc:824] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:38.338019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16305
I20260928 22:47:38.338299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41583
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=36345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:38.339533 16347 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:47:38.339830 16347 tablet_bootstrap.cc:492] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap complete.
I20260928 22:47:38.340775 16347 ts_tablet_manager.cc:1397] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.009s
I20260928 22:47:38.342079 16347 raft_consensus.cc:348] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:38.342319 16347 raft_consensus.cc:729] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1f28c3c4534606b53ac0d66c300b24, State: Initialized, Role: FOLLOWER
I20260928 22:47:38.342460 16347 consensus_queue.cc:260] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } }
I20260928 22:47:38.342741 16347 ts_tablet_manager.cc:1428] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:38.349041 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.334795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38309 (local address 127.0.98.62:33439)
0928 22:47:38.334980 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.334989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.337085 (+ 2096us) server_negotiation.cc:407] Connection header received
0928 22:47:38.337745 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.337751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.337796 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.337939 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:38.338861 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.340193 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.341665 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.342281 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.342474 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.345239 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:38.345243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:38.345244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:38.345255 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:38.346952 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.347531 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.347534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.347535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.347645 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.347979 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.347980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.347981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.348246 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:38.348324 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.348702 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.348895 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260928 22:47:38.349076 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.334740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:38309)
0928 22:47:38.336850 (+ 2110us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.336902 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.337527 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.338097 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.338102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.338459 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:38.338719 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.338726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.340322 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.340323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.341560 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.341563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.342601 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.342602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.342754 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.343321 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:38.343333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:38.345070 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:38.347055 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.347068 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.347071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.347424 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.347712 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.347713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.347713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.347871 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.348317 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:38.348325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:38.348505 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.348624 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.348849 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":103,"threads_started":1}
I20260928 22:47:38.349812 16455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:38.349928 16455 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:38.350111 16455 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:38.350850 16267 ts_manager.cc:194] Registered new tserver with Master: 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141)
I20260928 22:47:38.352164 16267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38309
I20260928 22:47:38.353063 16455 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
I20260928 22:47:38.356364 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.351615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.1:34141)
0928 22:47:38.352759 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.352763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.352772 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:47:38.352804 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.352806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.352955 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.353025 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:38.353228 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.353972 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.355485 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.355984 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.356047 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.356085 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.356274 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.356306 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":67,"threads_started":1}
I20260928 22:47:38.356396 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.351552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34141 (local address 127.0.0.1:42316)
0928 22:47:38.352511 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.352545 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.352611 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.353094 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.353097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.353100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:38.353169 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.353171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.354043 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.354044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.355372 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.355376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.356062 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.356063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.356170 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.356185 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.356254 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.356316 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":97,"threads_started":1}
I20260928 22:47:38.356871 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b8d4a97ba4411fbd49c28d19beddf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42316
I20260928 22:47:38.357985 16464 tablet_replica.cc:330] stopping tablet replica
I20260928 22:47:38.358104 16464 raft_consensus.cc:2229] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:38.358176 16464 raft_consensus.cc:2258] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:38.358467 16464 ts_tablet_manager.cc:1905] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:38.360906 16464 ts_tablet_manager.cc:1918] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:38.360980 16464 log.cc:1196] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal/wals/04b8d4a97ba4411fbd49c28d19beddf4
I20260928 22:47:38.361183 16464 ts_tablet_manager.cc:1939] T 04b8d4a97ba4411fbd49c28d19beddf4 P 8f1f28c3c4534606b53ac0d66c300b24: Deleting consensus metadata
I20260928 22:47:38.361701 16254 catalog_manager.cc:4841] TS 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141): tablet 04b8d4a97ba4411fbd49c28d19beddf4 (table test-table [id=52351d481e814c7a8dd4b50c26a98b21]) successfully deleted
W20260928 22:47:38.423213 16460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:38.423429 16460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:38.423463 16460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:38.424681 16460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:38.424757 16460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:38.424799 16460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:47:38.426338 16460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36345
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:38.426582 16460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:38.426801 16460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:38.431622 16469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.431632 16470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.431660 16472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:38.431809 16460 server_base.cc:1034] running on GCE node
I20260928 22:47:38.432021 16460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:38.432250 16460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:38.433414 16460 hybrid_clock.cc:648] HybridClock initialized: now 1790635658433408 us; error 52 us; skew 500 ppm
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:38.435838 16460 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:47:38.436240 16460 webserver.cc:460] Webserver started at http://127.0.98.2:36345/ using document root <none> and password file <none>
I20260928 22:47:38.436410 16460 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:38.436487 16460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:38.437794 16460 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:38.438403 16479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.438555 16460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:47:38.438637 16460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "db31495757d74840bb382f49e9d13c0a"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "9f60187207dcd2a186e0bc47fc054ce6"
server_key_iv: "eba294e939a62a9b1682fa67f41dfe18"
server_key_version: "encryptionkey@0"
I20260928 22:47:38.438956 16460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:38.455235 16460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:38.457398 16460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:38.457521 16460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:38.457774 16460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:38.458067 16460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:38.458101 16460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.458123 16460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:38.458134 16460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.463836 16460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41583
I20260928 22:47:38.463902 16592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41583 every 8 connection(s)
I20260928 22:47:38.464232 16460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:38.473470 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.465083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37379 (local address 127.0.98.62:33439)
0928 22:47:38.465160 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.465162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.465284 (+ 122us) server_negotiation.cc:407] Connection header received
0928 22:47:38.465744 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.465745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.465755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.465815 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:38.466740 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.467241 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.468377 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.468616 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.468682 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.470870 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:38.470872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:38.470872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:38.470879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:38.472126 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.472543 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.472544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.472545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.472605 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.472880 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.472880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.472881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.472995 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:38.473047 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.473296 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.473400 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:47:38.473568 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.464993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:37379)
0928 22:47:38.465151 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.465196 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.465669 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.465880 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.465883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.466462 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:38.466662 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.466667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.467342 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.467343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.468200 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.468203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.468738 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.468740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.468854 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.469384 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:38.469393 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:38.470768 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:38.472210 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.472216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.472218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.472456 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.472686 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.472687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.472688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.472827 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.473045 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:38.473048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:38.473166 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.473245 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.473373 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 22:47:38.473963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16460
I20260928 22:47:38.474083 16593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:38.474184 16593 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:38.474283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45845
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=40569
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:38.474391 16593 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:38.474807 16267 ts_manager.cc:194] Registered new tserver with Master: db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
I20260928 22:47:38.475178 16267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37379
W20260928 22:47:38.554160 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:38.554378 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:38.554402 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:38.555557 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:38.555598 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:38.555622 16596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:47:38.557077 16596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40569
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:38.557293 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:38.557499 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:38.562217 16601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.562223 16604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:38.562234 16602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:38.562429 16596 server_base.cc:1034] running on GCE node
I20260928 22:47:38.562641 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:38.562880 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:38.564005 16596 hybrid_clock.cc:648] HybridClock initialized: now 1790635658563993 us; error 31 us; skew 500 ppm
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:38.566344 16596 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:47:38.566735 16596 webserver.cc:460] Webserver started at http://127.0.98.3:40569/ using document root <none> and password file <none>
I20260928 22:47:38.566926 16596 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:38.566980 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:38.568332 16596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:38.568900 16611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.569056 16596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.569136 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "b2bace48c44a46d6b773f3de81c73f3d"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "7b7a9d0c5b4719c26e3bf7934e8c46ad"
server_key_iv: "4651a40b0f468a83425736b24455f067"
server_key_version: "encryptionkey@0"
I20260928 22:47:38.569458 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:38.590371 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:38.592468 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:38.592610 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:38.592880 16596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:38.593190 16596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:38.593223 16596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.593253 16596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:38.593269 16596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:38.598960 16596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45845
I20260928 22:47:38.599009 16724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45845 every 8 connection(s)
I20260928 22:47:38.599368 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:47:38 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:47:38.608103 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.600082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46809 (local address 127.0.98.62:33439)
0928 22:47:38.600144 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.600146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.600321 (+ 175us) server_negotiation.cc:407] Connection header received
0928 22:47:38.600816 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.600817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.600827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.600893 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:38.601798 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.602278 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.603283 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.603478 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.603529 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.605706 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:38.605708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:38.605708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:38.605716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:38.606912 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.607325 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.607327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.607327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.607387 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.607595 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.607596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.607596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.607700 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:38.607748 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.607959 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.608026 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:47:38.608258 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.600059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:46809)
0928 22:47:38.600202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.600243 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.600738 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.600950 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.600954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.601522 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:47:38.601721 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.601726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.602388 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.602389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.603195 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.603198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.603565 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.603566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.603678 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.604141 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:47:38.604149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:47:38.605611 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:47:38.606996 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.607003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.607005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.607233 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.607449 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:47:38.607449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:47:38.607450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:47:38.607548 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:47:38.607744 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:47:38.607747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:47:38.607864 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.607942 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.608064 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 22:47:38.608831 16725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33439
I20260928 22:47:38.608932 16725 heartbeater.cc:464] Registering TS with master...
I20260928 22:47:38.609076 16725 heartbeater.cc:511] Master 127.0.98.62:33439 requested a full tablet report, sending...
I20260928 22:47:38.609428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16596
I20260928 22:47:38.609490 16267 ts_manager.cc:194] Registered new tserver with Master: b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845)
I20260928 22:47:38.609962 16267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46809
I20260928 22:47:38.615445 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.609787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.0.98.62:33439)
0928 22:47:38.609820 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.609822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.609992 (+ 170us) server_negotiation.cc:407] Connection header received
0928 22:47:38.610032 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.610033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.610050 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.610097 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:38.610637 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.611097 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.611811 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.612017 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.612082 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.613227 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:38.613228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:38.613229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:38.613235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:38.614413 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.614710 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.614711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.614711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.614763 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.614917 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.614918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.614919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.615025 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:38.615195 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.615231 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.615301 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:47:38.616132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:47:38.618677 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.616440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.0.98.62:33439)
0928 22:47:38.616487 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.616488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.616571 (+ 83us) server_negotiation.cc:407] Connection header received
0928 22:47:38.616630 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.616631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.616661 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.616710 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:47:38.616842 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.617298 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.618051 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.618245 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.618305 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.618476 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:47:38.618519 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 22:47:38.618630 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54464: token expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 22:47:38.618731 16458 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54464: token expired
W20260928 22:47:38.619516 16194 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33439)
I20260928 22:47:38.625459 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.619827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.98.62:33439)
0928 22:47:38.619861 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.619862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.619870 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:47:38.619888 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.619888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.619897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.619950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:47:38.620511 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.621071 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.621803 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.621980 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.622032 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.623038 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:47:38.623040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:47:38.623040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:47:38.623047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:47:38.624291 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.624649 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.624651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.624651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.624710 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:47:38.624971 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:47:38.624972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:47:38.624972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:47:38.625129 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:47:38.625218 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.625317 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.625387 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:47:38.626657 16267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:47:38.628237 16267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:47:38.630198 16372 tablet_service.cc:1467] Processing CreateTablet for tablet b1af36a2d9e945ff8b4342195f97351a (DEFAULT_TABLE table=test-table [id=77b7d5414a7e4ec0a2c736b8c6af0d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:38.630405 16372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af36a2d9e945ff8b4342195f97351a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:38.631659 16347 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap starting.
I20260928 22:47:38.631984 16347 tablet_bootstrap.cc:654] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:38.632474 16347 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: No bootstrap required, opened a new log
I20260928 22:47:38.632517 16347 ts_tablet_manager.cc:1397] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:47:38.632618 16347 raft_consensus.cc:348] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.632682 16347 raft_consensus.cc:374] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:38.632771 16347 raft_consensus.cc:729] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1f28c3c4534606b53ac0d66c300b24, State: Initialized, Role: FOLLOWER
I20260928 22:47:38.632817 16347 consensus_queue.cc:260] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.632910 16347 ts_tablet_manager.cc:1428] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:47:38.634186 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.629944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45845 (local address 127.0.0.1:43454)
0928 22:47:38.629988 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.629998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.630031 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.630454 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.630455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.630456 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:38.630502 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.630504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.631894 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.631894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.633051 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.633053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.633967 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.633968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.634035 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.634039 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.634066 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.634114 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:47:38.634268 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.629979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.0.1:52612)
0928 22:47:38.630113 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.630120 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.630156 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.630649 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.630650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.630652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:38.630694 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.630695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.631982 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.631983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.633053 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.633054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.633942 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.633943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.634052 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.634057 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.634148 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.634195 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260928 22:47:38.634289 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.630222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.0.98.2:41583)
0928 22:47:38.630402 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.630406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.630417 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:47:38.630455 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.630457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.630471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.630551 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:38.630772 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.631912 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.633141 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.633868 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.633941 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.634006 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.634171 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.634214 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260928 22:47:38.634572 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.630087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.0.98.3:45845)
0928 22:47:38.630279 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.630284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.630294 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:47:38.630334 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.630336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.630352 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.630452 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:38.630558 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.631804 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.633144 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.633876 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.634356 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.634433 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.634457 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.634495 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260928 22:47:38.634974 16527 tablet_service.cc:1467] Processing CreateTablet for tablet b1af36a2d9e945ff8b4342195f97351a (DEFAULT_TABLE table=test-table [id=77b7d5414a7e4ec0a2c736b8c6af0d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:38.635164 16659 tablet_service.cc:1467] Processing CreateTablet for tablet b1af36a2d9e945ff8b4342195f97351a (DEFAULT_TABLE table=test-table [id=77b7d5414a7e4ec0a2c736b8c6af0d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:47:38.635260 16527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af36a2d9e945ff8b4342195f97351a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:38.635459 16659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1af36a2d9e945ff8b4342195f97351a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:47:38.637996 16734 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Bootstrap starting.
I20260928 22:47:38.638098 16735 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Bootstrap starting.
I20260928 22:47:38.638622 16734 tablet_bootstrap.cc:654] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:38.638705 16735 tablet_bootstrap.cc:654] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Neither blocks nor log segments found. Creating new log.
I20260928 22:47:38.639102 16734 log.cc:824] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:38.639189 16735 log.cc:824] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:38.639837 16734 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: No bootstrap required, opened a new log
I20260928 22:47:38.639894 16734 ts_tablet_manager.cc:1397] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:47:38.641110 16735 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: No bootstrap required, opened a new log
I20260928 22:47:38.641178 16735 ts_tablet_manager.cc:1397] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:47:38.641644 16734 raft_consensus.cc:348] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.641824 16734 raft_consensus.cc:374] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:38.641865 16734 raft_consensus.cc:729] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db31495757d74840bb382f49e9d13c0a, State: Initialized, Role: FOLLOWER
I20260928 22:47:38.641994 16734 consensus_queue.cc:260] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.642278 16734 ts_tablet_manager.cc:1428] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:47:38.642297 16593 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
I20260928 22:47:38.643138 16735 raft_consensus.cc:348] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.643306 16735 raft_consensus.cc:374] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:47:38.643347 16735 raft_consensus.cc:729] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2bace48c44a46d6b773f3de81c73f3d, State: Initialized, Role: FOLLOWER
I20260928 22:47:38.643464 16735 consensus_queue.cc:260] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.643680 16735 ts_tablet_manager.cc:1428] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:47:38.643718 16725 heartbeater.cc:503] Master 127.0.98.62:33439 was elected leader, sending a full tablet report...
W20260928 22:47:38.714850 16594 tablet.cc:2369] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:47:38.803360 16739 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:47:38.803572 16739 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.803977 16739 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583), 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141)
I20260928 22:47:38.807124 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.804394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54285 (local address 127.0.98.1:34141)
0928 22:47:38.804470 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.804472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.804573 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:47:38.804629 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.804630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.804643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.804707 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:38.804949 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.805446 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806444 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.806810 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.806882 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.807015 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.807051 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:47:38.807158 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.804185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34141 (local address 127.0.98.3:54285)
0928 22:47:38.804493 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.804502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.804573 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.804776 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.804777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.804781 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:38.804878 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.804880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.805518 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.805520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.806362 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.806365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.806883 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.806957 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.806962 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.807005 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.807047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":18,"thread_start_us":91,"threads_started":1}
I20260928 22:47:38.807181 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.804301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53083 (local address 127.0.98.2:41583)
0928 22:47:38.804363 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.804365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.804828 (+ 463us) server_negotiation.cc:407] Connection header received
0928 22:47:38.804984 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.804986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.804999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.805062 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:47:38.805195 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.805645 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806492 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.806869 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806925 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.806935 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.807077 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.807115 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:47:38.807379 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.804177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.98.3:53083)
0928 22:47:38.804279 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.804292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:38.804831 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:38.805071 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:47:38.805072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:47:38.805073 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:47:38.805134 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.805135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.805710 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.805711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.806408 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:47:38.806410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.806935 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.806936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:47:38.807010 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.807013 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:47:38.807057 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:47:38.807105 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 22:47:38.807696 16547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af36a2d9e945ff8b4342195f97351a" candidate_uuid: "b2bace48c44a46d6b773f3de81c73f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db31495757d74840bb382f49e9d13c0a" is_pre_election: true
I20260928 22:47:38.807684 16409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af36a2d9e945ff8b4342195f97351a" candidate_uuid: "b2bace48c44a46d6b773f3de81c73f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1f28c3c4534606b53ac0d66c300b24" is_pre_election: true
I20260928 22:47:38.807834 16547 raft_consensus.cc:2454] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2bace48c44a46d6b773f3de81c73f3d in term 0.
I20260928 22:47:38.807829 16409 raft_consensus.cc:2454] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2bace48c44a46d6b773f3de81c73f3d in term 0.
I20260928 22:47:38.808122 16615 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24, b2bace48c44a46d6b773f3de81c73f3d; no voters:
I20260928 22:47:38.808300 16739 raft_consensus.cc:2781] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:47:38.808382 16739 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:47:38.808399 16739 raft_consensus.cc:3037] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:38.809157 16739 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.809273 16739 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [CANDIDATE]: Term 1 election: Requested vote from peers db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583), 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141)
I20260928 22:47:38.809440 16547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af36a2d9e945ff8b4342195f97351a" candidate_uuid: "b2bace48c44a46d6b773f3de81c73f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db31495757d74840bb382f49e9d13c0a"
I20260928 22:47:38.809504 16409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1af36a2d9e945ff8b4342195f97351a" candidate_uuid: "b2bace48c44a46d6b773f3de81c73f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1f28c3c4534606b53ac0d66c300b24"
I20260928 22:47:38.809530 16547 raft_consensus.cc:3037] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:38.809592 16409 raft_consensus.cc:3037] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:47:38.810274 16547 raft_consensus.cc:2454] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2bace48c44a46d6b773f3de81c73f3d in term 1.
I20260928 22:47:38.810442 16613 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a; no voters:
I20260928 22:47:38.810534 16739 raft_consensus.cc:2781] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:47:38.810622 16409 raft_consensus.cc:2454] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2bace48c44a46d6b773f3de81c73f3d in term 1.
I20260928 22:47:38.810664 16739 raft_consensus.cc:686] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 1 LEADER]: Becoming Leader. State: Replica: b2bace48c44a46d6b773f3de81c73f3d, State: Running, Role: LEADER
I20260928 22:47:38.810752 16739 consensus_queue.cc:237] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:38.811643 16267 catalog_manager.cc:5495] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b2bace48c44a46d6b773f3de81c73f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } health_report { overall_health: UNKNOWN } } }
W20260928 22:47:38.834863 16456 tablet.cc:2369] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:47:38.849982 16726 tablet.cc:2369] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:47:38.861677 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.859028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.0.98.3:45845)
0928 22:47:38.859082 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:47:38.859084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:47:38.859092 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:47:38.859128 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:47:38.859128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:47:38.859147 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:47:38.859212 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:47:38.859467 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.859953 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.860663 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:47:38.860977 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:47:38.861050 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:47:38.861277 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:47:38.861410 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:47:38.861488 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:47:38.861560 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:47:38.861581 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:47:38.863445 16547 raft_consensus.cc:1261] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Refusing update from remote peer b2bace48c44a46d6b773f3de81c73f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:38.863459 16409 raft_consensus.cc:1261] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Refusing update from remote peer b2bace48c44a46d6b773f3de81c73f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:47:38.863828 16741 consensus_queue.cc:1035] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:38.863955 16739 consensus_queue.cc:1035] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:47:38.868744 16266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:38.868865 16266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:47:38.869580 16266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Sending DeleteTablet for 3 replicas of tablet b1af36a2d9e945ff8b4342195f97351a
I20260928 22:47:38.869884 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet b1af36a2d9e945ff8b4342195f97351a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42316
I20260928 22:47:38.869889 16659 tablet_service.cc:1514] Processing DeleteTablet for tablet b1af36a2d9e945ff8b4342195f97351a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43454
I20260928 22:47:38.870113 16527 tablet_service.cc:1514] Processing DeleteTablet for tablet b1af36a2d9e945ff8b4342195f97351a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52612
I20260928 22:47:38.870165 16755 tablet_replica.cc:330] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: stopping tablet replica
I20260928 22:47:38.870185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16305
I20260928 22:47:38.870214 16755 raft_consensus.cc:2229] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:38.870260 16755 raft_consensus.cc:2258] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:38.870311 16756 tablet_replica.cc:330] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: stopping tablet replica
I20260928 22:47:38.870869 16756 raft_consensus.cc:2229] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:47:38.870982 16756 raft_consensus.cc:2258] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:38.871430 16756 ts_tablet_manager.cc:1905] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:38.873019 16754 tablet_replica.cc:330] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: stopping tablet replica
I20260928 22:47:38.873160 16754 raft_consensus.cc:2229] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:47:38.877029 16756 ts_tablet_manager.cc:1918] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:38.877059 16754 raft_consensus.cc:2258] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:47:38.877134 16756 log.cc:1196] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/b1af36a2d9e945ff8b4342195f97351a
W20260928 22:47:38.877525 16254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34141 (error 108)
I20260928 22:47:38.877590 16756 ts_tablet_manager.cc:1939] T b1af36a2d9e945ff8b4342195f97351a P db31495757d74840bb382f49e9d13c0a: Deleting consensus metadata
I20260928 22:47:38.877950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16460
I20260928 22:47:38.879009 16754 ts_tablet_manager.cc:1905] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:47:38.879475 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.879316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34141 (local address 127.0.0.1:42324)
0928 22:47:38.879383 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.879437 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:47:38.879598 16254 catalog_manager.cc:4584] TS 8f1f28c3c4534606b53ac0d66c300b24 (127.0.98.1:34141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1af36a2d9e945ff8b4342195f97351a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34141: connect: Connection refused (error 111)
I20260928 22:47:38.881189 16754 ts_tablet_manager.cc:1918] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:47:38.881260 16754 log.cc:1196] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/b1af36a2d9e945ff8b4342195f97351a
I20260928 22:47:38.881511 16754 ts_tablet_manager.cc:1939] T b1af36a2d9e945ff8b4342195f97351a P b2bace48c44a46d6b773f3de81c73f3d: Deleting consensus metadata
I20260928 22:47:38.881943 16251 catalog_manager.cc:4841] TS b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): tablet b1af36a2d9e945ff8b4342195f97351a (table test-table [id=77b7d5414a7e4ec0a2c736b8c6af0d44]) successfully deleted
I20260928 22:47:38.883740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16596
I20260928 22:47:38.883896 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:38.883778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.0.1:52628)
0928 22:47:38.883831 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:38.883861 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:47:38.888473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16234
I20260928 22:47:58.893716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--webserver_interface=127.0.98.62
--webserver_port=34277
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:47:58.976173 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:58.976397 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:58.976420 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:58.977643 16758 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:47:58.977682 16758 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:47:58.977694 16758 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:47:58.977705 16758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:47:58.977723 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:58.977735 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:58.977751 16758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:47:58.977767 16758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:47:58.979336 16758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:58.979552 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:58.979734 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:47:58.984632 16758 server_base.cc:1034] running on GCE node
W20260928 22:47:58.984678 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:58.984634 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:58.984611 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:58.985126 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:58.985396 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:58.986570 16758 hybrid_clock.cc:648] HybridClock initialized: now 1790635678986531 us; error 54 us; skew 500 ppm
Sep 28 22:47:58 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:58.992889 16758 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:47:58.993355 16758 webserver.cc:460] Webserver started at http://127.0.98.62:34277/ using document root <none> and password file <none>
I20260928 22:47:58.993570 16758 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:58.993615 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:58.995038 16758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:58.995679 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:58.995854 16758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:47:58.995945 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7aace59e680e454bb30006a99890f577"
format_stamp: "Formatted at 2026-09-28 22:47:16 on dist-test-slave-pxr1"
server_key: "871100bb6e030b6bd0e6563839a03d4a"
server_key_iv: "456a638e3b95cc25db81199bfb809d16"
server_key_version: "encryptionkey@0"
I20260928 22:47:58.996300 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:59.023528 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:59.025811 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:59.025933 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:59.029661 16758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33439
I20260928 22:47:59.029700 16825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33439 every 8 connection(s)
I20260928 22:47:59.030058 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:47:59.031507 16826 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:47:59.032173 16826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap starting.
I20260928 22:47:59.038983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16758
I20260928 22:47:59.039292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34141
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=39591
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:59.040433 16826 log.cc:824] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:59.042663 16826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:47:59.042944 16826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Bootstrap complete.
I20260928 22:47:59.045099 16826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:59.045339 16826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aace59e680e454bb30006a99890f577, State: Initialized, Role: FOLLOWER
I20260928 22:47:59.045503 16826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:59.045580 16826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:47:59.045629 16826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:47:59.045675 16826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:47:59.046657 16826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:59.046754 16826 leader_election.cc:302] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aace59e680e454bb30006a99890f577; no voters:
I20260928 22:47:59.046914 16826 leader_election.cc:288] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:47:59.047009 16830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:47:59.047205 16826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:47:59.047236 16830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [term 3 LEADER]: Becoming Leader. State: Replica: 7aace59e680e454bb30006a99890f577, State: Running, Role: LEADER
I20260928 22:47:59.047390 16830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } }
I20260928 22:47:59.047747 16830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aace59e680e454bb30006a99890f577. Latest consensus state: current_term: 3 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:59.047822 16830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:59.048179 16831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7aace59e680e454bb30006a99890f577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace59e680e454bb30006a99890f577" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33439 } } }
I20260928 22:47:59.048250 16831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577 [sys.catalog]: This master's current role is: LEADER
I20260928 22:47:59.048394 16838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:47:59.049201 16838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:47:59.049299 16838 catalog_manager.cc:1259] Loaded cluster ID: 5b5adc09fc804609aeef80545e21c2a6
I20260928 22:47:59.049328 16838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:47:59.050567 16838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:47:59.050750 16838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Loaded TSK: 0
I20260928 22:47:59.050797 16838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7aace59e680e454bb30006a99890f577: Loaded TSK: 1
F20260928 22:47:59.051018 16838 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f13105ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f13105ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f13105ab4de google::LogMessage::Flush() at ??:0
@ 0x7f13105aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1312d18104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1312d15d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f13185051cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f131850548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1318505594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f13184cbc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f13184d5ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f13124790d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f131246e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f13132d26db start_thread at ??:0
@ 0x7f130f0a961f clone at ??:0
W20260928 22:47:59.140247 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:59.140477 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:59.140503 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:59.141806 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:59.141857 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:59.141886 16828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:47:59.143452 16828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39591
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:59.143709 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:59.143935 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:59.150524 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:59.150498 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:59.150746 16828 server_base.cc:1034] running on GCE node
W20260928 22:47:59.150839 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:59.151156 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:59.151419 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:59.152592 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790635679152585 us; error 47 us; skew 500 ppm
Sep 28 22:47:59 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:59.155098 16828 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:47:59.155584 16828 webserver.cc:460] Webserver started at http://127.0.98.1:39591/ using document root <none> and password file <none>
I20260928 22:47:59.155788 16828 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:59.155836 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:59.157274 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:59.157954 16877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.158211 16828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:59.158305 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "8f1f28c3c4534606b53ac0d66c300b24"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "28343f55d19dfd2c74ec534f899587b8"
server_key_iv: "1bf46d7faca3f38dec91374114e34998"
server_key_version: "encryptionkey@0"
I20260928 22:47:59.158669 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:59.174780 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:59.177008 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:59.177157 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:59.177381 16828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:59.177810 16884 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:47:59.179888 16828 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:47:59.179950 16828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:47:59.179989 16828 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:47:59.180626 16828 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:47:59.180660 16828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:47:59.180783 16884 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap starting.
I20260928 22:47:59.187340 16828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34141
I20260928 22:47:59.187505 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34141 every 8 connection(s)
I20260928 22:47:59.187767 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:47:59.191488 16884 log.cc:824] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Log is configured to *not* fsync() on all Append() calls
I20260928 22:47:59.194464 16884 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:47:59.194597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16828
I20260928 22:47:59.194793 16884 tablet_bootstrap.cc:492] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Bootstrap complete.
I20260928 22:47:59.194862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41583
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=36345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:59.195993 16884 ts_tablet_manager.cc:1397] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260928 22:47:59.197556 16884 raft_consensus.cc:348] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:59.197921 16884 raft_consensus.cc:729] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1f28c3c4534606b53ac0d66c300b24, State: Initialized, Role: FOLLOWER
I20260928 22:47:59.198146 16884 consensus_queue.cc:260] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:47:59.198455 16884 ts_tablet_manager.cc:1428] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
W20260928 22:47:59.254520 16994 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:47:59.188372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:52659)
0928 22:47:59.188536 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.188579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:47:59.189075 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:47:59.254296 (+ 65221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260928 22:47:59.254810 16992 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33439: Client connection negotiation failed: client connection to 127.0.98.62:33439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:47:59.255193 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.255062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:49859)
0928 22:47:59.255118 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.255165 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:47:59.255548 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.255459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:49619)
0928 22:47:59.255502 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.255527 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:47:59.255661 16992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:47:59.279740 16996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:59.279964 16996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:59.279990 16996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:59.281276 16996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:59.281327 16996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:59.281355 16996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:47:59.282904 16996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36345
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:59.283157 16996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:59.283378 16996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:47:59.288266 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:59.288332 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:59.288332 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:59.288573 16996 server_base.cc:1034] running on GCE node
I20260928 22:47:59.288787 16996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:59.289081 16996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:59.290243 16996 hybrid_clock.cc:648] HybridClock initialized: now 1790635679290229 us; error 47 us; skew 500 ppm
Sep 28 22:47:59 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:59.293761 16996 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:47:59.294274 16996 webserver.cc:460] Webserver started at http://127.0.98.2:36345/ using document root <none> and password file <none>
I20260928 22:47:59.294477 16996 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:59.294517 16996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:59.295818 16996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:47:59.296437 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.296638 16996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:47:59.296749 16996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "db31495757d74840bb382f49e9d13c0a"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "9f60187207dcd2a186e0bc47fc054ce6"
server_key_iv: "eba294e939a62a9b1682fa67f41dfe18"
server_key_version: "encryptionkey@0"
I20260928 22:47:59.297113 16996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:59.320401 16996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:59.322700 16996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:59.322914 16996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:59.323146 16996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:59.323410 16996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:59.323438 16996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.323472 16996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:59.323489 16996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.329144 16996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41583
I20260928 22:47:59.329224 17125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41583 every 8 connection(s)
I20260928 22:47:59.329530 16996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:47:59.331045 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.330384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:52519)
0928 22:47:59.330596 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.330675 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
W20260928 22:47:59.331329 17126 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33439: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
I20260928 22:47:59.331382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 16996
I20260928 22:47:59.331641 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.331543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:39189)
0928 22:47:59.331589 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.331619 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 22:47:59.331668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45845
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33439
--webserver_port=40569
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:47:59.332074 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.331987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:54805)
0928 22:47:59.332027 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.332054 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:47:59.332237 17126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:47:59.413720 17129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:47:59.413945 17129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:47:59.413970 17129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:47:59.415190 17129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:47:59.415237 17129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:47:59.415264 17129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:47:59.416869 17129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40569
--tserver_master_addrs=127.0.98.62:33439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:47:59.417078 17129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:47:59.417266 17129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:47:59.422147 17129 server_base.cc:1034] running on GCE node
W20260928 22:47:59.422091 17135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:59.422109 17137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:47:59.422091 17134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:47:59.422518 17129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:47:59.422757 17129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:47:59.423895 17129 hybrid_clock.cc:648] HybridClock initialized: now 1790635679423882 us; error 27 us; skew 500 ppm
Sep 28 22:47:59 dist-test-slave-pxr1 krb5kdc[15686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:47:59.426203 17129 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:47:59.426604 17129 webserver.cc:460] Webserver started at http://127.0.98.3:40569/ using document root <none> and password file <none>
I20260928 22:47:59.426806 17129 fs_manager.cc:359] Metadata directory not provided
I20260928 22:47:59.426854 17129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:47:59.428056 17129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:47:59.428532 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.428692 17129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.428794 17129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "b2bace48c44a46d6b773f3de81c73f3d"
format_stamp: "Formatted at 2026-09-28 22:47:17 on dist-test-slave-pxr1"
server_key: "7b7a9d0c5b4719c26e3bf7934e8c46ad"
server_key_iv: "4651a40b0f468a83425736b24455f067"
server_key_version: "encryptionkey@0"
I20260928 22:47:59.429126 17129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:47:59.460443 17129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:47:59.462683 17129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:47:59.462826 17129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:47:59.463065 17129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:47:59.463362 17129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:47:59.463395 17129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.463426 17129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:47:59.463455 17129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:47:59.469094 17129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45845
I20260928 22:47:59.469148 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45845 every 8 connection(s)
I20260928 22:47:59.469475 17129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:47:59.470455 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.470103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:36145)
0928 22:47:59.470252 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.470321 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260928 22:47:59.470827 17258 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33439 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33439: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
I20260928 22:47:59.471110 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.471026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:40037)
0928 22:47:59.471067 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.471093 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 22:47:59.471321 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:47:59.471265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:48979)
0928 22:47:59.471294 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 22:47:59.471310 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 22:47:59.471398 17258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:47:59.476970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17129
I20260928 22:48:00.256361 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.256099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:32991)
0928 22:48:00.256245 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:00.256290 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260928 22:48:00.332892 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.332562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:46897)
0928 22:48:00.332711 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:00.332811 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260928 22:48:00.471952 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.471727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:34265)
0928 22:48:00.471859 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:00.471900 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260928 22:48:00.928851 17265 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:00.928978 17265 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:48:00.929307 17265 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845), db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
Sep 28 22:48:00 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 22:48:00 dist-test-slave-pxr1 krb5kdc[15686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 22:48:00.939332 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.929484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41583 (local address 127.0.98.1:52341)
0928 22:48:00.929606 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:00.929616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:00.929667 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:00.930540 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:00.930544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:00.930920 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:00.931124 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:00.931131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.932220 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.932221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:00.933130 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:00.933132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.933688 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:00.933815 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:00.934297 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:00.934307 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:00.935778 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:00.937707 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:00.937715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:00.937717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:00.937993 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:00.938242 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:00.938243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:00.938348 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:00.938773 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:00.938780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:00.938931 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 22:48:00.939063 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 22:48:00.939227 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":59,"thread_start_us":54,"threads_started":1}
I20260928 22:48:00.939424 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.929576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52341 (local address 127.0.98.2:41583)
0928 22:48:00.929776 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:00.929784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:00.929802 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:48:00.930286 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:00.930289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:00.930351 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:00.930483 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:00.931229 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.932129 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933212 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.933606 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933695 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:00.935935 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:00.935941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:00.935941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:00.935953 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:00.937625 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938070 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:00.938073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:00.938073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:00.938178 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938409 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:00.938410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:00.938411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:00.938697 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:00.938757 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:00.939159 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 22:48:00.939314 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260928 22:48:00.939528 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.929584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39657 (local address 127.0.98.3:45845)
0928 22:48:00.929765 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:00.929775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:00.929805 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:48:00.930338 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:00.930341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:00.930390 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:00.930515 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:00.931243 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.932111 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933204 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.933627 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933727 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:00.936350 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:00.936354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:00.936354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:00.936366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:00.938072 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938414 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:00.938417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:00.938417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:00.938497 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938696 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:00.938697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:00.938697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:00.939012 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:00.939087 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:00.939294 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:48:00.939419 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260928 22:48:00.940009 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:00.929477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45845 (local address 127.0.98.1:39657)
0928 22:48:00.929594 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:00.929606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:00.929665 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:00.930541 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:00.930544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:00.930921 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:00.931126 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:00.931131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.932204 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.932205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:00.933128 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:00.933131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:00.933701 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:00.933702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:00.933902 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:00.934451 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:00.934455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:00.936222 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:00.938146 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:00.938147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:00.938343 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:00.938553 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:00.938553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:00.938553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:00.938610 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:00.939093 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:00.939094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:00.939144 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:48:00.939225 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:48:00.939317 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":49,"thread_start_us":52,"threads_started":1}
W20260928 22:48:00.940687 16879 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
W20260928 22:48:00.940804 16878 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
I20260928 22:48:00.940876 16878 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24; no voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a
I20260928 22:48:00.941020 17265 raft_consensus.cc:2726] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:48:01.257081 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:01.256932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:45865)
0928 22:48:01.257003 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:01.257040 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:48:01.333640 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:01.333355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:53209)
0928 22:48:01.333529 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:01.333587 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260928 22:48:01.472677 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:01.472434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:55507)
0928 22:48:01.472592 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:01.472635 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260928 22:48:02.257952 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:02.257683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:37137)
0928 22:48:02.257855 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:02.257903 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260928 22:48:02.334378 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:02.334169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:39491)
0928 22:48:02.334285 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:02.334325 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 22:48:02.473322 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:02.473130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:60091)
0928 22:48:02.473239 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:02.473278 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260928 22:48:02.948164 17275 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:02.948294 17275 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:48:02.948473 17275 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845), db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
W20260928 22:48:02.948947 16878 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
W20260928 22:48:02.949023 16879 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
I20260928 22:48:02.949062 16879 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24; no voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a
I20260928 22:48:02.949137 17275 raft_consensus.cc:2726] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:48:03.258754 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:03.258509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:41061)
0928 22:48:03.258647 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:03.258689 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 22:48:03.335078 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:03.334846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:59053)
0928 22:48:03.334970 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:03.335015 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260928 22:48:03.473984 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:03.473779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:50693)
0928 22:48:03.473892 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:03.473926 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260928 22:48:04.259594 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:04.259371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:47437)
0928 22:48:04.259501 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:04.259546 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 22:48:04.335745 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:04.335521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:36475)
0928 22:48:04.335642 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:04.335687 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 22:48:04.474638 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:04.474434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:57397)
0928 22:48:04.474547 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:04.474593 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260928 22:48:04.585060 17282 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:04.585161 17282 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:48:04.585332 17282 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845), db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
W20260928 22:48:04.585698 16878 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
W20260928 22:48:04.585789 16879 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
I20260928 22:48:04.585860 16879 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24; no voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a
I20260928 22:48:04.585932 17282 raft_consensus.cc:2726] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:48:05.260332 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:05.260101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:46115)
0928 22:48:05.260242 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:05.260283 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 22:48:05.336437 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:05.336223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:47377)
0928 22:48:05.336344 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:05.336388 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260928 22:48:05.475294 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:05.475095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:49397)
0928 22:48:05.475211 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:05.475251 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 22:48:06.261086 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:06.260870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:41693)
0928 22:48:06.260993 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:06.261034 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 22:48:06.337188 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:06.336961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:33027)
0928 22:48:06.337098 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:06.337140 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260928 22:48:06.475917 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:06.475713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:37657)
0928 22:48:06.475835 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:06.475875 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 22:48:07.261816 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:07.261586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:50499)
0928 22:48:07.261725 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:07.261767 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260928 22:48:07.337905 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:07.337671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:35123)
0928 22:48:07.337805 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:07.337848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 22:48:07.476583 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:07.476325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:51693)
0928 22:48:07.476489 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:07.476534 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260928 22:48:08.262466 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:08.262252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:43987)
0928 22:48:08.262383 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:08.262418 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260928 22:48:08.338677 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:08.338436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:39141)
0928 22:48:08.338583 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:08.338628 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260928 22:48:08.477259 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:08.477056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:44279)
0928 22:48:08.477169 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:08.477212 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260928 22:48:08.489791 17295 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:08.489912 17295 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:48:08.490090 17295 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845), db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
W20260928 22:48:08.490551 16878 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
W20260928 22:48:08.490644 16879 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
I20260928 22:48:08.490705 16879 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24; no voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a
I20260928 22:48:08.490769 17295 raft_consensus.cc:2726] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:48:09.263254 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:09.262960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:48843)
0928 22:48:09.263159 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:09.263206 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260928 22:48:09.339427 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:09.339177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:58833)
0928 22:48:09.339356 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:09.339382 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260928 22:48:09.477892 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:09.477681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:38123)
0928 22:48:09.477809 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:09.477845 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260928 22:48:10.263993 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:10.263729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:50167)
0928 22:48:10.263893 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:10.263936 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260928 22:48:10.340235 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:10.339973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:42349)
0928 22:48:10.340135 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:10.340187 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260928 22:48:10.478686 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:10.478416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:51175)
0928 22:48:10.478579 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:10.478628 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260928 22:48:11.264653 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:11.264421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:40619)
0928 22:48:11.264560 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:11.264598 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260928 22:48:11.341015 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:11.340777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:45395)
0928 22:48:11.340926 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:11.340968 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260928 22:48:11.479501 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:11.479228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:41139)
0928 22:48:11.479401 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:11.479456 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260928 22:48:12.265316 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:12.265071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:52997)
0928 22:48:12.265231 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:12.265269 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260928 22:48:12.341763 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:12.341521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:57767)
0928 22:48:12.341671 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:12.341714 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260928 22:48:12.480227 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:12.479979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:43359)
0928 22:48:12.480125 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:12.480180 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260928 22:48:12.798051 17308 raft_consensus.cc:482] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:12.798177 17308 raft_consensus.cc:504] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2bace48c44a46d6b773f3de81c73f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45845 } } peers { permanent_uuid: "db31495757d74840bb382f49e9d13c0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41583 } } peers { permanent_uuid: "8f1f28c3c4534606b53ac0d66c300b24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34141 } }
I20260928 22:48:12.798348 17308 leader_election.cc:288] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845), db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583)
W20260928 22:48:12.798717 16879 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db31495757d74840bb382f49e9d13c0a (127.0.98.2:41583): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
W20260928 22:48:12.798805 16878 leader_election.cc:341] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2bace48c44a46d6b773f3de81c73f3d (127.0.98.3:45845): Not found: Tablet not found: b1af36a2d9e945ff8b4342195f97351a
I20260928 22:48:12.798882 16878 leader_election.cc:302] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1f28c3c4534606b53ac0d66c300b24; no voters: b2bace48c44a46d6b773f3de81c73f3d, db31495757d74840bb382f49e9d13c0a
I20260928 22:48:12.798959 17308 raft_consensus.cc:2726] T b1af36a2d9e945ff8b4342195f97351a P 8f1f28c3c4534606b53ac0d66c300b24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:48:13.265936 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:13.265737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:45939)
0928 22:48:13.265861 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:13.265890 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 22:48:13.342459 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:13.342242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:60935)
0928 22:48:13.342371 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:13.342412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260928 22:48:13.481086 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:13.480761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:52425)
0928 22:48:13.480922 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:13.480965 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260928 22:48:14.266692 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:14.266450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.1:57261)
0928 22:48:14.266598 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:14.266644 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260928 22:48:14.343205 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:14.342997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.2:51669)
0928 22:48:14.343119 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:14.343159 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 22:48:14.481984 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:14.481711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33439 (local address 127.0.98.3:57663)
0928 22:48:14.481881 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:14.481935 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260928 22:48:14.486894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16828
I20260928 22:48:14.491125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 16996
I20260928 22:48:14.497088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17129
2026-09-28T22:48:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58682 ms)
[----------] 30 tests from SecurityITest (225604 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:14 dist-test-slave-pxr1 krb5kdc[17317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:14 dist-test-slave-pxr1 krb5kdc[17317](info): set up 2 sockets
Sep 28 22:48:14 dist-test-slave-pxr1 krb5kdc[17317](info): commencing operation
krb5kdc: starting...
W20260928 22:48:15.444133 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:15 dist-test-slave-pxr1 krb5kdc[17317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:15.556366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:15.635921 17330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:15.636154 17330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:15.636191 17330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:15.637420 17330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:15.637471 17330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:15.637490 17330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:15.637508 17330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:15.637524 17330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:15.639062 17330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:15.639317 17330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:15.639533 17330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:15.644477 17338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:15.644492 17336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:15.644565 17330 server_base.cc:1034] running on GCE node
W20260928 22:48:15.644496 17335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:15.644938 17330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:15.645242 17330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:15.646387 17330 hybrid_clock.cc:648] HybridClock initialized: now 1790635695646373 us; error 32 us; skew 500 ppm
Sep 28 22:48:15 dist-test-slave-pxr1 krb5kdc[17317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:15.648752 17330 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:15.649176 17330 webserver.cc:460] Webserver started at http://127.0.98.62:39999/ using document root <none> and password file <none>
I20260928 22:48:15.649386 17330 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:15.649427 17330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:15.649556 17330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:15.650611 17330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "a8fab4c871ef4cb09fb71bfd08a8102a"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "2d7230a12336a788d89d53458d3d882c"
server_key_iv: "49892546b40212e07a6abdb89fcca3a8"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.650930 17330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8fab4c871ef4cb09fb71bfd08a8102a"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "2d7230a12336a788d89d53458d3d882c"
server_key_iv: "49892546b40212e07a6abdb89fcca3a8"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.652173 17330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:15.652925 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:15.653080 17330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:15.653170 17330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "a8fab4c871ef4cb09fb71bfd08a8102a"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "2d7230a12336a788d89d53458d3d882c"
server_key_iv: "49892546b40212e07a6abdb89fcca3a8"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.653240 17330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:15.674022 17330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:15.676007 17330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:15.676146 17330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:15.679822 17330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33945
I20260928 22:48:15.679883 17397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33945 every 8 connection(s)
I20260928 22:48:15.680192 17330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:15.681098 17398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:15.681164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17330
I20260928 22:48:15.681265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:15.681489 392 external_mini_cluster.cc:1428] Setting key 07581a8b091c8da2f2b7796fa717a206
I20260928 22:48:15.684265 17398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: Bootstrap starting.
I20260928 22:48:15.684998 17398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:15.685287 17398 log.cc:824] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:15.686005 17398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: No bootstrap required, opened a new log
Sep 28 22:48:15 dist-test-slave-pxr1 krb5kdc[17317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:15.688364 17398 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:48:15.688557 17398 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:15.688597 17398 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fab4c871ef4cb09fb71bfd08a8102a, State: Initialized, Role: FOLLOWER
I20260928 22:48:15.688755 17398 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:48:15.688848 17398 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:15.688890 17398 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:15.688937 17398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:15.689607 17398 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:48:15.689715 17398 leader_election.cc:302] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8fab4c871ef4cb09fb71bfd08a8102a; no voters:
I20260928 22:48:15.689908 17398 leader_election.cc:288] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:15.689949 17403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:15.690125 17403 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [term 1 LEADER]: Becoming Leader. State: Replica: a8fab4c871ef4cb09fb71bfd08a8102a, State: Running, Role: LEADER
I20260928 22:48:15.690189 17398 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:15.690232 17403 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } }
I20260928 22:48:15.690788 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8fab4c871ef4cb09fb71bfd08a8102a. Latest consensus state: current_term: 1 leader_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } } }
I20260928 22:48:15.690945 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:15.691183 17412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:15.691251 17404 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fab4c871ef4cb09fb71bfd08a8102a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33945 } } }
I20260928 22:48:15.691365 17404 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:15.691826 17412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:15.692790 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:15.682018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.98.62:33945)
0928 22:48:15.682177 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:15.682185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:15.682211 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:48:15.682646 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:15.682652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:15.682700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:15.682816 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:15.683551 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.684453 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.685487 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.685990 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.686097 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:15.688680 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:15.688685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:15.688685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:15.688699 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:15.690620 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:15.691033 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:15.691035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:15.691036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:15.691143 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:15.691358 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:15.691359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:15.691359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:15.691719 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:15.691864 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:15.692392 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 22:48:15.692550 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260928 22:48:15.693491 17412 catalog_manager.cc:1347] Generated new cluster ID: e39b9d903bc3462d9df93c995407f02d
I20260928 22:48:15.693542 17412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:15.700802 17412 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:15.701295 17412 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:15.710122 17412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: Generated new TSK 0
I20260928 22:48:15.710309 17412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:48:15.741695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:15.843420 17424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:15.843636 17424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:15.843658 17424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:15.844825 17424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:15.844861 17424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:15.844885 17424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:15.846331 17424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:15.846562 17424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:15.846762 17424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:15.851432 17430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:15.851464 17432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:15.851511 17429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:15.851668 17424 server_base.cc:1034] running on GCE node
I20260928 22:48:15.851882 17424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:15.852084 17424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:15.853211 17424 hybrid_clock.cc:648] HybridClock initialized: now 1790635695853201 us; error 31 us; skew 500 ppm
Sep 28 22:48:15 dist-test-slave-pxr1 krb5kdc[17317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:15.855582 17424 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:48:15.856005 17424 webserver.cc:460] Webserver started at http://127.0.98.1:36753/ using document root <none> and password file <none>
I20260928 22:48:15.856210 17424 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:15.856256 17424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:15.856376 17424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:15.857467 17424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "86274bba5f403b21ea197ea2262041b1"
server_key_iv: "e502a176aaf5afade608642824c0b0d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.857782 17424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "86274bba5f403b21ea197ea2262041b1"
server_key_iv: "e502a176aaf5afade608642824c0b0d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.859010 17424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:48:15.859660 17439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:15.859807 17424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:15.859885 17424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2"
format_stamp: "Formatted at 2026-09-28 22:48:15 on dist-test-slave-pxr1"
server_key: "86274bba5f403b21ea197ea2262041b1"
server_key_iv: "e502a176aaf5afade608642824c0b0d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:15.859951 17424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:15.878558 17424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:15.880491 17424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:15.880630 17424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:15.880887 17424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:15.881198 17424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:15.881232 17424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:15.881260 17424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:15.881284 17424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:15.886828 17424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43449
I20260928 22:48:15.886883 17552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43449 every 8 connection(s)
I20260928 22:48:15.887209 17424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:48:15 dist-test-slave-pxr1 krb5kdc[17317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:15.896996 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:15.887941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43871 (local address 127.0.98.62:33945)
0928 22:48:15.888019 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:15.888021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:15.888205 (+ 184us) server_negotiation.cc:407] Connection header received
0928 22:48:15.889227 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:15.889228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:15.889239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:15.889304 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:15.889943 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.890477 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.891714 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.891978 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.892026 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:15.894244 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:15.894245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:15.894245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:15.894254 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:15.895515 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:15.895975 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:15.895976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:15.895976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:15.896036 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:15.896336 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:15.896337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:15.896338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:15.896466 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:15.896527 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:15.896843 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 22:48:15.896926 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:48:15.897282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17424
I20260928 22:48:15.897380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:15.897554 392 external_mini_cluster.cc:1428] Setting key ac0d6190756a110bc03354880c0a6b9b
I20260928 22:48:15.897716 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:15.887885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33945 (local address 127.0.98.1:43871)
0928 22:48:15.888055 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:15.888098 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:15.889105 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:15.889362 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:15.889366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:15.889627 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:15.889834 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:15.889839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.890573 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.890574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:15.891630 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:15.891633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:15.892047 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:15.892048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:15.892163 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:15.892625 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:15.892632 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:15.894142 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:15.895620 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:15.895628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:15.895631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:15.895884 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:15.896122 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:15.896123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:15.896124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:15.896252 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:15.896523 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:15.896527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:15.896656 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 22:48:15.896769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:48:15.896911 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260928 22:48:15.898351 17553 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33945
I20260928 22:48:15.898459 17553 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:15.898640 17553 heartbeater.cc:511] Master 127.0.98.62:33945 requested a full tablet report, sending...
I20260928 22:48:15.899173 17361 ts_manager.cc:194] Registered new tserver with Master: c1e4062e0a6f4b7db97a517ba3ddbda2 (127.0.98.1:43449)
I20260928 22:48:15.899927 17361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43871
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:48:15.923025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:16.005502 17558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.005841 17558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.005919 17558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.007252 17558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.007300 17558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.007339 17558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:16.008982 17558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.009233 17558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.009456 17558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.014593 17564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.014602 17566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.014611 17558 server_base.cc:1034] running on GCE node
W20260928 22:48:16.014596 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.015020 17558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.015232 17558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.016390 17558 hybrid_clock.cc:648] HybridClock initialized: now 1790635696016378 us; error 28 us; skew 500 ppm
Sep 28 22:48:16 dist-test-slave-pxr1 krb5kdc[17317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:16.018899 17558 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:48:16.019315 17558 webserver.cc:460] Webserver started at http://127.0.98.2:39367/ using document root <none> and password file <none>
I20260928 22:48:16.019503 17558 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.019543 17558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.019635 17558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.020668 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "e51836645bf448299ea143be73885f43"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "fea1b266089e42a9feb914ece669d47c"
server_key_iv: "ffafb8874648528549609133520c8762"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.021041 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e51836645bf448299ea143be73885f43"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "fea1b266089e42a9feb914ece669d47c"
server_key_iv: "ffafb8874648528549609133520c8762"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.022245 17558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:16.022927 17573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.023088 17558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:16.023169 17558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "e51836645bf448299ea143be73885f43"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "fea1b266089e42a9feb914ece669d47c"
server_key_iv: "ffafb8874648528549609133520c8762"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.023226 17558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.037334 17558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.039181 17558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.039292 17558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.039475 17558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:16.039753 17558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:16.039790 17558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.039822 17558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:16.039844 17558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.045445 17558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40867
I20260928 22:48:16.045516 17686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40867 every 8 connection(s)
I20260928 22:48:16.045799 17558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:16.048871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17558
I20260928 22:48:16.048965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:16.049155 392 external_mini_cluster.cc:1428] Setting key d48b984c22b46883d4933ec6cc43fe56
Sep 28 22:48:16 dist-test-slave-pxr1 krb5kdc[17317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:16.055665 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.046532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43681 (local address 127.0.98.62:33945)
0928 22:48:16.046599 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.046601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.046794 (+ 193us) server_negotiation.cc:407] Connection header received
0928 22:48:16.047730 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.047732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.047757 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.047815 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.048392 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.049000 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.050319 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.050505 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.050555 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.052796 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.052798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.052798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:16.052806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.054338 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.054726 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.054727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.054727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.054791 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.055011 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.055012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.055012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.055158 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.055226 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.055460 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.055564 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:48:16.055773 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.046519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33945 (local address 127.0.98.2:43681)
0928 22:48:16.046679 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.046716 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.047603 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.047876 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.047879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.048128 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:16.048316 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.048320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.049083 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.049084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.050224 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.050228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.050581 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.050582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.050726 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.051272 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:16.051283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:16.052668 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:16.054424 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:16.054432 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:16.054434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:16.054656 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:16.054852 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:16.054852 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:16.054852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:16.054952 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:16.055216 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:16.055219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:16.055339 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.055422 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.055567 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260928 22:48:16.056346 17687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33945
I20260928 22:48:16.056444 17687 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:16.056640 17687 heartbeater.cc:511] Master 127.0.98.62:33945 requested a full tablet report, sending...
I20260928 22:48:16.057060 17361 ts_manager.cc:194] Registered new tserver with Master: e51836645bf448299ea143be73885f43 (127.0.98.2:40867)
I20260928 22:48:16.057439 17361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43681
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:48:16.076242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:16.158034 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.158243 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.158277 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.159472 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.159515 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.159552 17692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:16.161121 17692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.161350 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.161558 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.166340 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.166371 17692 server_base.cc:1034] running on GCE node
W20260928 22:48:16.166335 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.166380 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.166795 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.167034 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.168171 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790635696168156 us; error 29 us; skew 500 ppm
Sep 28 22:48:16 dist-test-slave-pxr1 krb5kdc[17317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:16.170572 17692 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:48:16.171022 17692 webserver.cc:460] Webserver started at http://127.0.98.3:35519/ using document root <none> and password file <none>
I20260928 22:48:16.171216 17692 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.171262 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.171379 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.172433 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "5abd3ed0cfda42e6a57ecca91f10f52f"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e1c9ec3ab0d282f7a9c3bf1e2c90d15c"
server_key_iv: "c1612827e7a01e973421b73072679a66"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.172804 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5abd3ed0cfda42e6a57ecca91f10f52f"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e1c9ec3ab0d282f7a9c3bf1e2c90d15c"
server_key_iv: "c1612827e7a01e973421b73072679a66"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.174060 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:16.174737 17707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.174868 17692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:16.174949 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "5abd3ed0cfda42e6a57ecca91f10f52f"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e1c9ec3ab0d282f7a9c3bf1e2c90d15c"
server_key_iv: "c1612827e7a01e973421b73072679a66"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.175016 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.203963 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.206154 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.206295 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.206517 17692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:16.206815 17692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:16.206853 17692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.206883 17692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:16.206907 17692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.212481 17692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41729
I20260928 22:48:16.212527 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41729 every 8 connection(s)
I20260928 22:48:16.212865 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:48:16 dist-test-slave-pxr1 krb5kdc[17317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:16.221477 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.213606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59401 (local address 127.0.98.62:33945)
0928 22:48:16.213658 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.213660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.213819 (+ 159us) server_negotiation.cc:407] Connection header received
0928 22:48:16.214733 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.214735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.214748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.214798 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.215402 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.215896 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.216934 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.217122 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.217174 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.219124 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.219125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.219126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:16.219133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.220241 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.220595 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.220596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.220597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.220648 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.220887 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.220888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.220888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.221005 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.221060 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.221305 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.221384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:48:16.221498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17692
I20260928 22:48:16.221593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:16.221655 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.213553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33945 (local address 127.0.98.3:59401)
0928 22:48:16.213700 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.213744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.214631 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.214868 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.214872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.215126 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:16.215315 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.215319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.215985 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.215986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.216844 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.216847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.217187 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.217188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.217312 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.217772 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:16.217780 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:16.219037 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:16.220313 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:16.220319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:16.220321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:16.220532 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:16.220700 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:16.220701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:16.220701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:16.220833 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:16.221059 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:16.221066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:16.221194 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.221279 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.221444 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260928 22:48:16.221766 392 external_mini_cluster.cc:1428] Setting key cbe3c6109af8a8dd83e9953406bafb76
I20260928 22:48:16.222267 17821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33945
I20260928 22:48:16.222370 17821 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:16.222548 17821 heartbeater.cc:511] Master 127.0.98.62:33945 requested a full tablet report, sending...
I20260928 22:48:16.222888 17362 ts_manager.cc:194] Registered new tserver with Master: 5abd3ed0cfda42e6a57ecca91f10f52f (127.0.98.3:41729)
I20260928 22:48:16.223239 17362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59401
I20260928 22:48:16.233318 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:16.239902 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.234595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.98.62:33945)
0928 22:48:16.234637 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.234639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.234750 (+ 111us) server_negotiation.cc:407] Connection header received
0928 22:48:16.234779 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.234780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.234789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.234841 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.235335 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.235809 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.236474 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.236645 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.236688 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.237835 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.237836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.237836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:16.237842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.239033 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.239324 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.239325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.239325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.239370 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:16.239495 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:16.239496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:16.239496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:16.239630 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.239700 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.239778 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.239834 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:16.241340 17362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:16.242918 17362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:16.248975 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.0.1:44742)
0928 22:48:16.245119 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.245159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.245221 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.245747 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.245748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.245750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.245813 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.245814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.246745 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.246746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.247867 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.247870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.248639 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.248640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.248753 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.248770 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.248836 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.248890 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
I20260928 22:48:16.248980 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.0.98.2:40867)
0928 22:48:16.245585 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.245589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.245599 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:48:16.245636 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.245638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.245653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.245881 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.245958 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.246650 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.247952 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.248565 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.248626 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.248685 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.248860 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.248901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":62,"threads_started":1}
I20260928 22:48:16.249398 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43449 (local address 127.0.0.1:53902)
0928 22:48:16.245358 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.245368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.245433 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.245625 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.245627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.245630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.245699 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.245701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.247157 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.247157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.248145 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.248147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.249168 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.249169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.249229 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.249233 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.249288 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.249327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":38,"thread_start_us":79,"threads_started":1}
I20260928 22:48:16.249414 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.1:43449)
0928 22:48:16.245358 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.245362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.245429 (+ 67us) server_negotiation.cc:407] Connection header received
0928 22:48:16.245517 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.245519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.245535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.245613 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.245810 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.247043 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.248324 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.249018 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.249072 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.249131 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.249305 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.249343 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260928 22:48:16.249842 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.0.1:59038)
0928 22:48:16.245540 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.245548 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.245598 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.245891 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.245892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.245894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.245942 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.245943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.247153 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.247154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.248181 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.248184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.249674 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.249675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.249731 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.249734 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.249766 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.249795 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":26,"thread_start_us":68,"threads_started":1}
I20260928 22:48:16.249853 17621 tablet_service.cc:1467] Processing CreateTablet for tablet aed53bcb5ff842408e6109c939661f47 (DEFAULT_TABLE table=test-table [id=9506427dc9c14b4dbedf34ec791f91f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.249953 17487 tablet_service.cc:1467] Processing CreateTablet for tablet aed53bcb5ff842408e6109c939661f47 (DEFAULT_TABLE table=test-table [id=9506427dc9c14b4dbedf34ec791f91f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.250232 17621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed53bcb5ff842408e6109c939661f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.250231 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed53bcb5ff842408e6109c939661f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.250265 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.245380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.0.98.3:41729)
0928 22:48:16.245538 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.245542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.245768 (+ 226us) server_negotiation.cc:407] Connection header received
0928 22:48:16.245808 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.245810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.245824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.245895 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.246015 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.247052 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.248268 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.249001 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.250055 (+ 1054us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.250127 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.250152 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.250193 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260928 22:48:16.250900 17755 tablet_service.cc:1467] Processing CreateTablet for tablet aed53bcb5ff842408e6109c939661f47 (DEFAULT_TABLE table=test-table [id=9506427dc9c14b4dbedf34ec791f91f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.251190 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed53bcb5ff842408e6109c939661f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.253727 17840 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Bootstrap starting.
I20260928 22:48:16.253753 17839 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Bootstrap starting.
I20260928 22:48:16.254381 17840 tablet_bootstrap.cc:654] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.254386 17839 tablet_bootstrap.cc:654] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.254940 17839 log.cc:824] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.254940 17840 log.cc:824] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.255283 17838 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: Bootstrap starting.
I20260928 22:48:16.255704 17839 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: No bootstrap required, opened a new log
I20260928 22:48:16.255774 17839 ts_tablet_manager.cc:1397] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:16.255906 17838 tablet_bootstrap.cc:654] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.255923 17840 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: No bootstrap required, opened a new log
I20260928 22:48:16.255976 17840 ts_tablet_manager.cc:1397] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:16.256417 17838 log.cc:824] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.257213 17838 tablet_bootstrap.cc:492] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: No bootstrap required, opened a new log
I20260928 22:48:16.257284 17838 ts_tablet_manager.cc:1397] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:16.257632 17839 raft_consensus.cc:348] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.257781 17839 raft_consensus.cc:374] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.257812 17839 raft_consensus.cc:729] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51836645bf448299ea143be73885f43, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.257795 17840 raft_consensus.cc:348] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.257975 17840 raft_consensus.cc:374] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.257949 17839 consensus_queue.cc:260] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.258016 17840 raft_consensus.cc:729] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abd3ed0cfda42e6a57ecca91f10f52f, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.258136 17840 consensus_queue.cc:260] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.258214 17687 heartbeater.cc:503] Master 127.0.98.62:33945 was elected leader, sending a full tablet report...
I20260928 22:48:16.258219 17839 ts_tablet_manager.cc:1428] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:16.258376 17840 ts_tablet_manager.cc:1428] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:16.258474 17821 heartbeater.cc:503] Master 127.0.98.62:33945 was elected leader, sending a full tablet report...
I20260928 22:48:16.259140 17838 raft_consensus.cc:348] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.259310 17838 raft_consensus.cc:374] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.259357 17838 raft_consensus.cc:729] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e4062e0a6f4b7db97a517ba3ddbda2, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.259472 17838 consensus_queue.cc:260] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.259694 17838 ts_tablet_manager.cc:1428] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:16.259730 17553 heartbeater.cc:503] Master 127.0.98.62:33945 was elected leader, sending a full tablet report...
W20260928 22:48:16.296348 17688 tablet.cc:2369] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:16.343842 17845 raft_consensus.cc:482] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:16.343976 17845 raft_consensus.cc:504] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.344300 17845 leader_election.cc:288] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e4062e0a6f4b7db97a517ba3ddbda2 (127.0.98.1:43449), e51836645bf448299ea143be73885f43 (127.0.98.2:40867)
I20260928 22:48:16.347232 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.344690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54235 (local address 127.0.98.2:40867)
0928 22:48:16.344776 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.344778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.344878 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:48:16.344912 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.344913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.344925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.344976 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.345116 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.345565 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346541 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.346917 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346978 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.346989 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.347126 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.347163 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 22:48:16.347246 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.344512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43449 (local address 127.0.98.3:40977)
0928 22:48:16.344574 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.344587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.344655 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.344827 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.344828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.344830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.344904 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.344906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.345595 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.345596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.346445 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.346448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.346987 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.347069 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.347071 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.347118 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.347160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":30}
I20260928 22:48:16.347241 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.344616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40977 (local address 127.0.98.1:43449)
0928 22:48:16.344665 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.344667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.344681 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:48:16.344710 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.344711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.344750 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.344807 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.344975 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.345506 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346543 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.346910 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.346980 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.347136 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.347176 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:48:16.347257 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.344512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40867 (local address 127.0.98.3:54235)
0928 22:48:16.344810 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.344819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.344872 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.344986 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.344987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.344988 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.345056 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.345058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.345641 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.345642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.346445 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.346447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.346993 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.346995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.347060 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.347063 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.347107 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.347148 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":65,"thread_start_us":113,"threads_started":1}
I20260928 22:48:16.347599 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed53bcb5ff842408e6109c939661f47" candidate_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" is_pre_election: true
I20260928 22:48:16.347592 17641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed53bcb5ff842408e6109c939661f47" candidate_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51836645bf448299ea143be73885f43" is_pre_election: true
I20260928 22:48:16.347756 17507 raft_consensus.cc:2454] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5abd3ed0cfda42e6a57ecca91f10f52f in term 0.
I20260928 22:48:16.347765 17641 raft_consensus.cc:2454] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5abd3ed0cfda42e6a57ecca91f10f52f in term 0.
I20260928 22:48:16.347994 17709 leader_election.cc:302] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5abd3ed0cfda42e6a57ecca91f10f52f, e51836645bf448299ea143be73885f43; no voters:
I20260928 22:48:16.348155 17845 raft_consensus.cc:2781] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:16.348206 17845 raft_consensus.cc:482] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:16.348222 17845 raft_consensus.cc:3037] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.348901 17845 raft_consensus.cc:504] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.349021 17845 leader_election.cc:288] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [CANDIDATE]: Term 1 election: Requested vote from peers c1e4062e0a6f4b7db97a517ba3ddbda2 (127.0.98.1:43449), e51836645bf448299ea143be73885f43 (127.0.98.2:40867)
I20260928 22:48:16.349195 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed53bcb5ff842408e6109c939661f47" candidate_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2"
I20260928 22:48:16.349198 17641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed53bcb5ff842408e6109c939661f47" candidate_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51836645bf448299ea143be73885f43"
I20260928 22:48:16.349284 17507 raft_consensus.cc:3037] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.349284 17641 raft_consensus.cc:3037] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.350129 17507 raft_consensus.cc:2454] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5abd3ed0cfda42e6a57ecca91f10f52f in term 1.
I20260928 22:48:16.350128 17641 raft_consensus.cc:2454] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5abd3ed0cfda42e6a57ecca91f10f52f in term 1.
I20260928 22:48:16.350317 17711 leader_election.cc:302] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5abd3ed0cfda42e6a57ecca91f10f52f, c1e4062e0a6f4b7db97a517ba3ddbda2; no voters:
I20260928 22:48:16.350404 17845 raft_consensus.cc:2781] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:16.350540 17845 raft_consensus.cc:686] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 1 LEADER]: Becoming Leader. State: Replica: 5abd3ed0cfda42e6a57ecca91f10f52f, State: Running, Role: LEADER
I20260928 22:48:16.350625 17845 consensus_queue.cc:237] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } }
I20260928 22:48:16.351434 17362 catalog_manager.cc:5495] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5abd3ed0cfda42e6a57ecca91f10f52f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abd3ed0cfda42e6a57ecca91f10f52f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 } health_report { overall_health: UNKNOWN } } }
I20260928 22:48:16.363725 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.361477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.98.3:41729)
0928 22:48:16.361537 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.361539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.361547 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:16.361582 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.361583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.361598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.361658 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:16.361788 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.362297 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.363073 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.363267 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.363332 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.363463 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:16.363564 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:16.363620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.363633 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.363651 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:48:16.365479 17507 raft_consensus.cc:1261] T aed53bcb5ff842408e6109c939661f47 P c1e4062e0a6f4b7db97a517ba3ddbda2 [term 1 FOLLOWER]: Refusing update from remote peer 5abd3ed0cfda42e6a57ecca91f10f52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:16.365485 17641 raft_consensus.cc:1261] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 1 FOLLOWER]: Refusing update from remote peer 5abd3ed0cfda42e6a57ecca91f10f52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:16.365779 17845 consensus_queue.cc:1035] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e4062e0a6f4b7db97a517ba3ddbda2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:16.365931 17848 consensus_queue.cc:1035] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51836645bf448299ea143be73885f43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:16.368777 17852 mvcc.cc:205] Tried to move back new op lower bound from 7334443812310515712 to 7334443812252995584. Current Snapshot: MvccSnapshot[applied={T|T < 7334443812310515712}]
I20260928 22:48:16.369292 17853 mvcc.cc:205] Tried to move back new op lower bound from 7334443812310515712 to 7334443812252995584. Current Snapshot: MvccSnapshot[applied={T|T < 7334443812310515712 or (T in {7334443812310515712})}]
I20260928 22:48:16.370656 17361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:16.370750 17361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:16.371428 17361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8fab4c871ef4cb09fb71bfd08a8102a: Sending DeleteTablet for 3 replicas of tablet aed53bcb5ff842408e6109c939661f47
I20260928 22:48:16.371762 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet aed53bcb5ff842408e6109c939661f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59038
I20260928 22:48:16.371771 17621 tablet_service.cc:1514] Processing DeleteTablet for tablet aed53bcb5ff842408e6109c939661f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44742
I20260928 22:48:16.371837 17487 tablet_service.cc:1514] Processing DeleteTablet for tablet aed53bcb5ff842408e6109c939661f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53902
I20260928 22:48:16.372027 17860 tablet_replica.cc:330] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: stopping tablet replica
I20260928 22:48:16.372124 17860 raft_consensus.cc:2229] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:16.372208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17424
I20260928 22:48:16.372246 17860 raft_consensus.cc:2258] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:16.373847 17860 ts_tablet_manager.cc:1905] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:16.374048 17861 tablet_replica.cc:330] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: stopping tablet replica
I20260928 22:48:16.374202 17861 raft_consensus.cc:2229] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:16.374374 17861 raft_consensus.cc:2258] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:16.374888 17861 ts_tablet_manager.cc:1905] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:16.375633 17860 ts_tablet_manager.cc:1918] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:16.375699 17860 log.cc:1196] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/aed53bcb5ff842408e6109c939661f47
I20260928 22:48:16.375923 17860 ts_tablet_manager.cc:1939] T aed53bcb5ff842408e6109c939661f47 P 5abd3ed0cfda42e6a57ecca91f10f52f: Deleting consensus metadata
I20260928 22:48:16.376302 17349 catalog_manager.cc:4841] TS 5abd3ed0cfda42e6a57ecca91f10f52f (127.0.98.3:41729): tablet aed53bcb5ff842408e6109c939661f47 (table test-table [id=9506427dc9c14b4dbedf34ec791f91f7]) successfully deleted
I20260928 22:48:16.376524 17861 ts_tablet_manager.cc:1918] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:16.376598 17861 log.cc:1196] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/aed53bcb5ff842408e6109c939661f47
I20260928 22:48:16.376891 17861 ts_tablet_manager.cc:1939] T aed53bcb5ff842408e6109c939661f47 P e51836645bf448299ea143be73885f43: Deleting consensus metadata
I20260928 22:48:16.377328 17347 catalog_manager.cc:4841] TS e51836645bf448299ea143be73885f43 (127.0.98.2:40867): tablet aed53bcb5ff842408e6109c939661f47 (table test-table [id=9506427dc9c14b4dbedf34ec791f91f7]) successfully deleted
W20260928 22:48:16.379581 17349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43449 (error 108)
I20260928 22:48:16.379784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17558
I20260928 22:48:16.380589 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.380468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43449 (local address 127.0.0.1:53908)
0928 22:48:16.380517 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.380554 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:48:16.380697 17349 catalog_manager.cc:4584] TS c1e4062e0a6f4b7db97a517ba3ddbda2 (127.0.98.1:43449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aed53bcb5ff842408e6109c939661f47: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43449: connect: Connection refused (error 111)
I20260928 22:48:16.384702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17692
I20260928 22:48:16.388993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17330
2026-09-28T22:48:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T22:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:16Z Disabled control of system clock
I20260928 22:48:16.414942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:16.494194 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.494424 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.494444 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.495626 17867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:16.495667 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.495678 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.495689 17867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:16.495713 17867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:16.497087 17867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.497316 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.497584 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.502053 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.502017 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.502182 17867 server_base.cc:1034] running on GCE node
W20260928 22:48:16.502122 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.502507 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.502780 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.503922 17867 hybrid_clock.cc:648] HybridClock initialized: now 1790635696503905 us; error 30 us; skew 500 ppm
I20260928 22:48:16.505084 17867 webserver.cc:460] Webserver started at http://127.0.98.62:44913/ using document root <none> and password file <none>
I20260928 22:48:16.505303 17867 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.505352 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.505474 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.506485 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "02802d8dba304ac79b91beba12c10a0e"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "6e1c5eb47ef956fb3beb2d914ecf294d"
server_key_iv: "3f59dcee6bb45d7bf58db34d6abcb578"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.506804 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "02802d8dba304ac79b91beba12c10a0e"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "6e1c5eb47ef956fb3beb2d914ecf294d"
server_key_iv: "3f59dcee6bb45d7bf58db34d6abcb578"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.507946 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:16.508594 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.508760 17867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:16.508844 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "02802d8dba304ac79b91beba12c10a0e"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "6e1c5eb47ef956fb3beb2d914ecf294d"
server_key_iv: "3f59dcee6bb45d7bf58db34d6abcb578"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.508910 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.520097 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.520397 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.520521 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.524251 17867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37857
I20260928 22:48:16.524286 17933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37857 every 8 connection(s)
I20260928 22:48:16.524606 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:16.525527 17934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.527725 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: Bootstrap starting.
I20260928 22:48:16.528259 17934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.528509 17934 log.cc:824] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.529160 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: No bootstrap required, opened a new log
I20260928 22:48:16.529191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17867
I20260928 22:48:16.529266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:16.529434 392 external_mini_cluster.cc:1428] Setting key 4436749e54d37cd111c107bb64e50367
I20260928 22:48:16.531073 17934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260928 22:48:16.531250 17934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.531286 17934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02802d8dba304ac79b91beba12c10a0e, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.531425 17934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260928 22:48:16.531505 17934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:16.531535 17934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:16.531571 17934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.532243 17934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260928 22:48:16.532351 17934 leader_election.cc:302] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02802d8dba304ac79b91beba12c10a0e; no voters:
I20260928 22:48:16.532500 17934 leader_election.cc:288] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:16.532549 17939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:16.532756 17934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:16.532764 17939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [term 1 LEADER]: Becoming Leader. State: Replica: 02802d8dba304ac79b91beba12c10a0e, State: Running, Role: LEADER
I20260928 22:48:16.532980 17939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260928 22:48:16.533353 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02802d8dba304ac79b91beba12c10a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260928 22:48:16.533424 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:16.533628 17941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02802d8dba304ac79b91beba12c10a0e. Latest consensus state: current_term: 1 leader_uuid: "02802d8dba304ac79b91beba12c10a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02802d8dba304ac79b91beba12c10a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260928 22:48:16.533903 17941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:16.534025 17948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:16.534548 17948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:16.536132 17948 catalog_manager.cc:1347] Generated new cluster ID: b72b56c652af4e97a7c320e5dc698381
I20260928 22:48:16.536182 17948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:16.536623 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.529916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.98.62:37857)
0928 22:48:16.530122 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.530130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.530163 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:48:16.531039 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.531048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.531126 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.531273 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.531496 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.533236 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.534339 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.534924 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.535054 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.535761 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.535765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.535765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:16.535970 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.536010 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:16.536025 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.536086 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.536347 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.536428 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260928 22:48:16.541533 17948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:16.542109 17948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:16.548786 17948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: Generated new TSK 0
I20260928 22:48:16.548966 17948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:16.555856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:16.635241 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.635456 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.635478 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.636691 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.636758 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.636801 17958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:16.638222 17958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.638458 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.638753 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.643483 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.643513 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.643741 17958 server_base.cc:1034] running on GCE node
W20260928 22:48:16.643491 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.644009 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.644246 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.645393 17958 hybrid_clock.cc:648] HybridClock initialized: now 1790635696645378 us; error 33 us; skew 500 ppm
I20260928 22:48:16.646644 17958 webserver.cc:460] Webserver started at http://127.0.98.1:41129/ using document root <none> and password file <none>
I20260928 22:48:16.646885 17958 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.646934 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.647060 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.648115 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f19a94285954107a8a8fb2a890e1eda"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "d260832dcb6fda9862880bb4cb630d75"
server_key_iv: "0263df39ad1439fd444920aaffc5db9e"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.648450 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f19a94285954107a8a8fb2a890e1eda"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "d260832dcb6fda9862880bb4cb630d75"
server_key_iv: "0263df39ad1439fd444920aaffc5db9e"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.649755 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:16.650491 17972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.650657 17958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:16.650748 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "9f19a94285954107a8a8fb2a890e1eda"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "d260832dcb6fda9862880bb4cb630d75"
server_key_iv: "0263df39ad1439fd444920aaffc5db9e"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.650815 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.662420 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.662689 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.662786 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.662967 17958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:16.663219 17958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:16.663261 17958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.663286 17958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:16.663302 17958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.668458 17958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38607
I20260928 22:48:16.668537 18085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38607 every 8 connection(s)
I20260928 22:48:16.668834 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:16.670310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 17958
I20260928 22:48:16.670424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:16.670610 392 external_mini_cluster.cc:1428] Setting key f84aa907e145f0b248a2219ee149275f
I20260928 22:48:16.671676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:48:16.674722 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.669522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59325 (local address 127.0.98.62:37857)
0928 22:48:16.669604 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.669606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.669785 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:48:16.670351 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.670352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.670364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.670404 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.671266 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.671868 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.673289 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.673591 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.673665 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.674468 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.674470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.674470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:16.674475 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.674493 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:16.674498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.674548 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.674609 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.674642 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:48:16.674898 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.669494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.1:59325)
0928 22:48:16.669671 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.669704 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.670252 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.670487 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.670491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.670501 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:16.671154 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.671162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.671958 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.671959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.673196 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.673199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.673669 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.673670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.673778 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.674317 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:16.674317 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:16.674346 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:16.674369 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:16.674559 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:16.674562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:16.674564 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.674609 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.674691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 22:48:16.675343 18086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260928 22:48:16.675459 18086 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:16.675669 18086 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260928 22:48:16.676245 17898 ts_manager.cc:194] Registered new tserver with Master: 9f19a94285954107a8a8fb2a890e1eda (127.0.98.1:38607)
I20260928 22:48:16.676930 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59325
W20260928 22:48:16.752813 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.753039 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.753062 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.754302 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.754352 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.754379 18089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:16.755951 18089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.756157 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.756417 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.761232 18095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.761247 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.761245 18097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.761303 18089 server_base.cc:1034] running on GCE node
I20260928 22:48:16.761706 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.761945 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.763113 18089 hybrid_clock.cc:648] HybridClock initialized: now 1790635696763096 us; error 25 us; skew 500 ppm
I20260928 22:48:16.764420 18089 webserver.cc:460] Webserver started at http://127.0.98.2:37497/ using document root <none> and password file <none>
I20260928 22:48:16.764663 18089 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.764734 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.764868 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.765949 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "0817b2507a0d46c9b4cc8da4759dce2a"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "821adc5ef00a57e2a12592cf28edda79"
server_key_iv: "221237149e8173c9f8b9055e5ee80538"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.766279 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0817b2507a0d46c9b4cc8da4759dce2a"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "821adc5ef00a57e2a12592cf28edda79"
server_key_iv: "221237149e8173c9f8b9055e5ee80538"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.767616 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:16.768309 18103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.768477 18089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:16.768558 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "0817b2507a0d46c9b4cc8da4759dce2a"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "821adc5ef00a57e2a12592cf28edda79"
server_key_iv: "221237149e8173c9f8b9055e5ee80538"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.768622 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.784694 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.785105 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.785233 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.785444 18089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:16.785766 18089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:16.785801 18089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.785830 18089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:16.785848 18089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.791380 18089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33877
I20260928 22:48:16.791437 18216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33877 every 8 connection(s)
I20260928 22:48:16.791769 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:16.796984 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.792531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49759 (local address 127.0.98.62:37857)
0928 22:48:16.792578 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.792580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.792749 (+ 169us) server_negotiation.cc:407] Connection header received
0928 22:48:16.793259 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.793260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.793269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.793325 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.794071 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.794570 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.795558 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.795795 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.795859 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.796695 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.796697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.796697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:16.796703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.796744 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:16.796748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.796818 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.796889 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.796919 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:16.797209 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.792426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.2:49759)
0928 22:48:16.792620 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.792662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.793177 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.793386 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.793390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.793398 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:16.793952 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.793958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.794645 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.794646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.795453 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.795456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.795873 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.795873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.795992 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.796534 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:16.796535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:16.796564 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:16.796588 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:16.796823 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:16.796826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:16.796829 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.796880 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.796966 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260928 22:48:16.797195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18089
I20260928 22:48:16.797305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:16.797466 392 external_mini_cluster.cc:1428] Setting key a830f674da207dc88b0fb8e502c7f053
I20260928 22:48:16.797777 18217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260928 22:48:16.797873 18217 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:16.798058 18217 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260928 22:48:16.798427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:48:16.798440 17898 ts_manager.cc:194] Registered new tserver with Master: 0817b2507a0d46c9b4cc8da4759dce2a (127.0.98.2:33877)
I20260928 22:48:16.798818 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49759
W20260928 22:48:16.877408 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:16.877610 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:16.877630 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:16.878790 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:16.878830 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:16.878858 18220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:16.880241 18220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:16.880470 18220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:16.880796 18220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:16.885704 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.885704 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:16.885710 18226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:16.885730 18220 server_base.cc:1034] running on GCE node
I20260928 22:48:16.886113 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:16.886353 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:16.887490 18220 hybrid_clock.cc:648] HybridClock initialized: now 1790635696887475 us; error 32 us; skew 500 ppm
I20260928 22:48:16.888751 18220 webserver.cc:460] Webserver started at http://127.0.98.3:43109/ using document root <none> and password file <none>
I20260928 22:48:16.888993 18220 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:16.889039 18220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:16.889165 18220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:16.890224 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "c60d134b867d4bbb89b3b02f448995ab"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e90568381cdaecd44455417d6cf5c1ca"
server_key_iv: "a820e3a912235b87a946e4452b7580fd"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.890547 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c60d134b867d4bbb89b3b02f448995ab"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e90568381cdaecd44455417d6cf5c1ca"
server_key_iv: "a820e3a912235b87a946e4452b7580fd"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.891767 18220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:16.892413 18234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.892701 18220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:16.892814 18220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "c60d134b867d4bbb89b3b02f448995ab"
format_stamp: "Formatted at 2026-09-28 22:48:16 on dist-test-slave-pxr1"
server_key: "e90568381cdaecd44455417d6cf5c1ca"
server_key_iv: "a820e3a912235b87a946e4452b7580fd"
server_key_version: "encryptionkey@0"
I20260928 22:48:16.892889 18220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:16.900540 18220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:16.900854 18220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:16.900978 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:16.901175 18220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:16.901489 18220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:16.901521 18220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.901542 18220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:16.901564 18220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:16.906934 18220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37825
I20260928 22:48:16.906998 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37825 every 8 connection(s)
I20260928 22:48:16.907320 18220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:16.912009 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.908005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38305 (local address 127.0.98.62:37857)
0928 22:48:16.908051 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.908054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.908253 (+ 199us) server_negotiation.cc:407] Connection header received
0928 22:48:16.908766 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.908768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.908780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.908835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.909459 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.909952 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.910842 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.911027 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.911070 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.911781 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.911782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.911783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:16.911787 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.911802 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:16.911806 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.911845 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.911911 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.911947 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:16.912182 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.907993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.3:38305)
0928 22:48:16.908155 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.908187 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.908659 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.908897 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.908900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.908909 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:16.909395 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.909399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.910010 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.910011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.910782 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.910784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.911081 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.911081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.911184 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.911642 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:16.911642 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:16.911669 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:16.911707 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:16.911847 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:16.911849 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:16.911851 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.911907 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.911985 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260928 22:48:16.912643 18348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260928 22:48:16.912751 18348 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:16.912961 18348 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260928 22:48:16.913031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18220
I20260928 22:48:16.913129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:16.913326 392 external_mini_cluster.cc:1428] Setting key c32f421236f0c6fe6e7f6b5746dfebe0
I20260928 22:48:16.913324 17898 ts_manager.cc:194] Registered new tserver with Master: c60d134b867d4bbb89b3b02f448995ab (127.0.98.3:37825)
I20260928 22:48:16.913789 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38305
I20260928 22:48:16.914151 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:16.918017 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.915498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.0.98.62:37857)
0928 22:48:16.915559 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.915560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.915631 (+ 71us) server_negotiation.cc:407] Connection header received
0928 22:48:16.915682 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.915682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.915692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.915734 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:16.915851 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.916307 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.917057 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.917232 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.917275 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.917836 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:16.917837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:16.917837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:16.917841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:16.917851 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:16.917854 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:16.917893 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.917943 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.917961 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:16.919481 17897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:16.921006 17897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:16.927382 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.922994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33877 (local address 127.0.0.1:46280)
0928 22:48:16.923649 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.923657 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.923698 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.923908 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.923909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.923911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.923967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.923969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.924947 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.924948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.926171 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.926174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.927141 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.927142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.927198 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927203 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.927250 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.927295 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":30,"thread_start_us":97,"threads_started":1}
I20260928 22:48:16.927462 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.923064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.0.98.3:37825)
0928 22:48:16.923205 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.923210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.923542 (+ 332us) server_negotiation.cc:407] Connection header received
0928 22:48:16.923628 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.923630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.923646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.923728 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.923989 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.925130 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.926164 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.926858 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.926939 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927032 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.927341 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.927381 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 22:48:16.927466 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.923237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.0.98.1:38607)
0928 22:48:16.923397 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.923400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.923525 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:48:16.923628 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.923630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.923646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.923728 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.923989 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.925143 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.926347 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.927017 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.927072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927149 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.927341 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.927391 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 22:48:16.927912 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.922987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37825 (local address 127.0.0.1:39516)
0928 22:48:16.923426 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.923467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.923545 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.923807 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.923810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.923812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.923910 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.923913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.925222 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.925223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.926092 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.926094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.927066 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.927067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.927144 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927158 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.927219 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.927272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":19,"spinlock_wait_cycles":768,"thread_start_us":71,"threads_started":1}
I20260928 22:48:16.928145 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.923238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.0.98.2:33877)
0928 22:48:16.923462 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.923464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.923713 (+ 249us) server_negotiation.cc:407] Connection header received
0928 22:48:16.923757 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.923759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.923774 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.923839 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.924037 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.924868 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.926264 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.926956 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.927023 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.928021 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.928067 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260928 22:48:16.927899 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.922994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38607 (local address 127.0.0.1:57104)
0928 22:48:16.923427 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.923467 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.923545 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.923815 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.923815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.923816 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.923910 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.923912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.925211 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.925212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.926277 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.926279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.927178 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.927179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.927231 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.927233 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.927271 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.927301 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260928 22:48:16.929196 18020 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0731295843c0874df7f609673b90 (DEFAULT_TABLE table=test-table [id=aecacd4ebe0d4abfaeb0b6c6884e76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.929235 18151 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0731295843c0874df7f609673b90 (DEFAULT_TABLE table=test-table [id=aecacd4ebe0d4abfaeb0b6c6884e76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.929579 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0731295843c0874df7f609673b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.929580 18151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0731295843c0874df7f609673b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.932094 18363 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Bootstrap starting.
I20260928 22:48:16.932085 18362 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: Bootstrap starting.
I20260928 22:48:16.932523 18282 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0731295843c0874df7f609673b90 (DEFAULT_TABLE table=test-table [id=aecacd4ebe0d4abfaeb0b6c6884e76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:16.932780 18363 tablet_bootstrap.cc:654] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.932826 18362 tablet_bootstrap.cc:654] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.932878 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0731295843c0874df7f609673b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:16.933094 18362 log.cc:824] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.933104 18363 log.cc:824] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.933849 18362 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: No bootstrap required, opened a new log
I20260928 22:48:16.933931 18362 ts_tablet_manager.cc:1397] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:48:16.936038 18366 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Bootstrap starting.
I20260928 22:48:16.936367 18362 raft_consensus.cc:348] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.936502 18362 raft_consensus.cc:374] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.936532 18362 raft_consensus.cc:729] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f19a94285954107a8a8fb2a890e1eda, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.936640 18366 tablet_bootstrap.cc:654] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:16.936646 18362 consensus_queue.cc:260] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.936902 18362 ts_tablet_manager.cc:1428] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:48:16.936990 18086 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
I20260928 22:48:16.936990 18366 log.cc:824] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:16.937301 18363 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: No bootstrap required, opened a new log
I20260928 22:48:16.937364 18363 ts_tablet_manager.cc:1397] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 22:48:16.937557 18366 tablet_bootstrap.cc:492] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: No bootstrap required, opened a new log
I20260928 22:48:16.937598 18366 ts_tablet_manager.cc:1397] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:16.939407 18363 raft_consensus.cc:348] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.939532 18363 raft_consensus.cc:374] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.939471 18366 raft_consensus.cc:348] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.939563 18363 raft_consensus.cc:729] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0817b2507a0d46c9b4cc8da4759dce2a, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.939601 18366 raft_consensus.cc:374] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:16.939638 18366 raft_consensus.cc:729] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60d134b867d4bbb89b3b02f448995ab, State: Initialized, Role: FOLLOWER
I20260928 22:48:16.939675 18363 consensus_queue.cc:260] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.939771 18366 consensus_queue.cc:260] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.939931 18363 ts_tablet_manager.cc:1428] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:48:16.939988 18217 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
I20260928 22:48:16.940025 18366 ts_tablet_manager.cc:1428] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:16.940140 18348 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
I20260928 22:48:16.973266 18370 raft_consensus.cc:482] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:16.973419 18370 raft_consensus.cc:504] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.973722 18370 leader_election.cc:288] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0817b2507a0d46c9b4cc8da4759dce2a (127.0.98.2:33877), 9f19a94285954107a8a8fb2a890e1eda (127.0.98.1:38607)
I20260928 22:48:16.976478 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.973950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54533 (local address 127.0.98.1:38607)
0928 22:48:16.973992 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.973994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.974006 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:16.974085 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.974086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.974098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.974149 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.974304 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.974793 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.975800 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.976160 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.976219 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.976228 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.976379 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.976411 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:16.976490 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.973889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38607 (local address 127.0.98.3:54533)
0928 22:48:16.973951 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.973962 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.974002 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.974159 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.974160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.974162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.974237 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.974238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.974867 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.974868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.975709 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.975711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.976250 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.976251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.976311 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.976314 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.976365 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.976401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":50}
I20260928 22:48:16.976478 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.973957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60663 (local address 127.0.98.2:33877)
0928 22:48:16.974013 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.974014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.974145 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:48:16.974183 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.974184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.974192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.974248 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:16.974390 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.974838 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.975795 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.976153 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.976212 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.976222 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.976353 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.976393 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:48:16.976521 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.973889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33877 (local address 127.0.98.3:60663)
0928 22:48:16.974067 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:16.974077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:16.974138 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:16.974253 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:16.974254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:16.974255 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:16.974329 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.974330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.974910 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.974911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.975706 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:16.975708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.976224 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.976225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:16.976289 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.976293 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:16.976332 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 22:48:16.976378 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":13,"thread_start_us":106,"threads_started":1}
I20260928 22:48:16.976836 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0731295843c0874df7f609673b90" candidate_uuid: "c60d134b867d4bbb89b3b02f448995ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" is_pre_election: true
I20260928 22:48:16.976835 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0731295843c0874df7f609673b90" candidate_uuid: "c60d134b867d4bbb89b3b02f448995ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f19a94285954107a8a8fb2a890e1eda" is_pre_election: true
I20260928 22:48:16.976986 18171 raft_consensus.cc:2454] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60d134b867d4bbb89b3b02f448995ab in term 0.
I20260928 22:48:16.976987 18040 raft_consensus.cc:2454] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60d134b867d4bbb89b3b02f448995ab in term 0.
I20260928 22:48:16.977186 18235 leader_election.cc:302] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0817b2507a0d46c9b4cc8da4759dce2a, c60d134b867d4bbb89b3b02f448995ab; no voters:
I20260928 22:48:16.977347 18370 raft_consensus.cc:2781] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:16.977391 18370 raft_consensus.cc:482] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:16.977411 18370 raft_consensus.cc:3037] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.978137 18370 raft_consensus.cc:504] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.978261 18370 leader_election.cc:288] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [CANDIDATE]: Term 1 election: Requested vote from peers 0817b2507a0d46c9b4cc8da4759dce2a (127.0.98.2:33877), 9f19a94285954107a8a8fb2a890e1eda (127.0.98.1:38607)
I20260928 22:48:16.978401 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0731295843c0874df7f609673b90" candidate_uuid: "c60d134b867d4bbb89b3b02f448995ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f19a94285954107a8a8fb2a890e1eda"
I20260928 22:48:16.978475 18040 raft_consensus.cc:3037] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.979116 18040 raft_consensus.cc:2454] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60d134b867d4bbb89b3b02f448995ab in term 1.
I20260928 22:48:16.979218 18171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0731295843c0874df7f609673b90" candidate_uuid: "c60d134b867d4bbb89b3b02f448995ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0817b2507a0d46c9b4cc8da4759dce2a"
I20260928 22:48:16.979278 18238 leader_election.cc:302] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f19a94285954107a8a8fb2a890e1eda, c60d134b867d4bbb89b3b02f448995ab; no voters:
I20260928 22:48:16.979295 18171 raft_consensus.cc:3037] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:16.979362 18370 raft_consensus.cc:2781] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:16.979525 18370 raft_consensus.cc:686] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 1 LEADER]: Becoming Leader. State: Replica: c60d134b867d4bbb89b3b02f448995ab, State: Running, Role: LEADER
I20260928 22:48:16.979616 18370 consensus_queue.cc:237] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } }
I20260928 22:48:16.980084 18171 raft_consensus.cc:2454] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60d134b867d4bbb89b3b02f448995ab in term 1.
I20260928 22:48:16.980535 17897 catalog_manager.cc:5495] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c60d134b867d4bbb89b3b02f448995ab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c60d134b867d4bbb89b3b02f448995ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60d134b867d4bbb89b3b02f448995ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 } health_report { overall_health: UNKNOWN } } }
I20260928 22:48:16.999182 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:16.997136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.0.98.3:37825)
0928 22:48:16.997173 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:16.997175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:16.997183 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:16.997256 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:16.997257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:16.997272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:16.997322 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:16.997416 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.997881 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.998567 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:16.998746 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:16.998790 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:16.998923 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:16.999019 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:16.999074 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:16.999086 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 22:48:16.999101 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:17.000903 18040 raft_consensus.cc:1261] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 1 FOLLOWER]: Refusing update from remote peer c60d134b867d4bbb89b3b02f448995ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:17.000907 18171 raft_consensus.cc:1261] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 1 FOLLOWER]: Refusing update from remote peer c60d134b867d4bbb89b3b02f448995ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:17.001240 18372 consensus_queue.cc:1035] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0817b2507a0d46c9b4cc8da4759dce2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:17.001376 18370 consensus_queue.cc:1035] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f19a94285954107a8a8fb2a890e1eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:17.004338 18376 mvcc.cc:205] Tried to move back new op lower bound from 7334443814913064960 to 7334443814829330432. Current Snapshot: MvccSnapshot[applied={T|T < 7334443814913064960}]
I20260928 22:48:17.004415 18377 mvcc.cc:205] Tried to move back new op lower bound from 7334443814913064960 to 7334443814829330432. Current Snapshot: MvccSnapshot[applied={T|T < 7334443814913064960}]
I20260928 22:48:17.005841 17897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:17.005957 17897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:17.006665 17897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02802d8dba304ac79b91beba12c10a0e: Sending DeleteTablet for 3 replicas of tablet e25f0731295843c0874df7f609673b90
I20260928 22:48:17.006980 18282 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0731295843c0874df7f609673b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:17 UTC) from {username='slave'} at 127.0.0.1:39516
I20260928 22:48:17.007004 18151 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0731295843c0874df7f609673b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:17 UTC) from {username='slave'} at 127.0.0.1:46280
I20260928 22:48:17.007109 18020 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0731295843c0874df7f609673b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:17 UTC) from {username='slave'} at 127.0.0.1:57104
I20260928 22:48:17.007220 18385 tablet_replica.cc:330] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: stopping tablet replica
I20260928 22:48:17.007323 18385 raft_consensus.cc:2229] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:17.007354 18387 tablet_replica.cc:330] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda: stopping tablet replica
I20260928 22:48:17.007443 18385 raft_consensus.cc:2258] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:17.007567 18387 raft_consensus.cc:2229] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:17.007643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17958
I20260928 22:48:17.007745 18387 raft_consensus.cc:2258] T e25f0731295843c0874df7f609673b90 P 9f19a94285954107a8a8fb2a890e1eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:17.008114 18385 ts_tablet_manager.cc:1905] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:17.009724 18386 tablet_replica.cc:330] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: stopping tablet replica
I20260928 22:48:17.009838 18386 raft_consensus.cc:2229] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:17.009948 18386 raft_consensus.cc:2258] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:17.010089 18385 ts_tablet_manager.cc:1918] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:17.010154 18385 log.cc:1196] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/e25f0731295843c0874df7f609673b90
I20260928 22:48:17.010386 18385 ts_tablet_manager.cc:1939] T e25f0731295843c0874df7f609673b90 P 0817b2507a0d46c9b4cc8da4759dce2a: Deleting consensus metadata
I20260928 22:48:17.010764 17882 catalog_manager.cc:4841] TS 0817b2507a0d46c9b4cc8da4759dce2a (127.0.98.2:33877): tablet e25f0731295843c0874df7f609673b90 (table test-table [id=aecacd4ebe0d4abfaeb0b6c6884e76d3]) successfully deleted
I20260928 22:48:17.011157 18386 ts_tablet_manager.cc:1905] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:17.012712 18386 ts_tablet_manager.cc:1918] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:17.012818 18386 log.cc:1196] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/e25f0731295843c0874df7f609673b90
I20260928 22:48:17.013036 18386 ts_tablet_manager.cc:1939] T e25f0731295843c0874df7f609673b90 P c60d134b867d4bbb89b3b02f448995ab: Deleting consensus metadata
I20260928 22:48:17.013535 17884 catalog_manager.cc:4841] TS c60d134b867d4bbb89b3b02f448995ab (127.0.98.3:37825): tablet e25f0731295843c0874df7f609673b90 (table test-table [id=aecacd4ebe0d4abfaeb0b6c6884e76d3]) successfully deleted
W20260928 22:48:17.014243 17885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38607 (error 108)
I20260928 22:48:17.014364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18089
I20260928 22:48:17.016219 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:17.016090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38607 (local address 127.0.0.1:57114)
0928 22:48:17.016145 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:17.016189 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:48:17.016341 17885 catalog_manager.cc:4584] TS 9f19a94285954107a8a8fb2a890e1eda (127.0.98.1:38607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e25f0731295843c0874df7f609673b90: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38607: connect: Connection refused (error 111)
I20260928 22:48:17.019145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18220
I20260928 22:48:17.023689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 17867
2026-09-28T22:48:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:17 dist-test-slave-pxr1 krb5kdc[18391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:17 dist-test-slave-pxr1 krb5kdc[18391](info): set up 2 sockets
Sep 28 22:48:17 dist-test-slave-pxr1 krb5kdc[18391](info): commencing operation
krb5kdc: starting...
W20260928 22:48:17.988210 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:18.103407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:18.181882 18404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:18.182093 18404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:18.182124 18404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:18.183312 18404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:18.183358 18404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:18.183375 18404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:18.183393 18404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:18.183408 18404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:18.184940 18404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:18.185189 18404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:18.185405 18404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:18.190011 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.190019 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.190019 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:18.190214 18404 server_base.cc:1034] running on GCE node
I20260928 22:48:18.190511 18404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:18.190809 18404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:18.191955 18404 hybrid_clock.cc:648] HybridClock initialized: now 1790635698191948 us; error 36 us; skew 500 ppm
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:18.194383 18404 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:18.194792 18404 webserver.cc:460] Webserver started at http://127.0.98.62:34109/ using document root <none> and password file <none>
I20260928 22:48:18.195001 18404 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:18.195053 18404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:18.195221 18404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:18.196275 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "a664227856c745d69408b1fab9e4b6fe"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "c68d5a6f2997940eb7ee92e6c3ddc047"
server_key_iv: "cd4101ec62ae24e92843e76785bc99f9"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.196589 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "a664227856c745d69408b1fab9e4b6fe"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "c68d5a6f2997940eb7ee92e6c3ddc047"
server_key_iv: "cd4101ec62ae24e92843e76785bc99f9"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.197919 18404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:18.198565 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.198724 18404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:18.198796 18404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "a664227856c745d69408b1fab9e4b6fe"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "c68d5a6f2997940eb7ee92e6c3ddc047"
server_key_iv: "cd4101ec62ae24e92843e76785bc99f9"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.198855 18404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:18.214586 18404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:18.216691 18404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:18.216902 18404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:18.220450 18404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42083
I20260928 22:48:18.220561 18471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42083 every 8 connection(s)
I20260928 22:48:18.220840 18404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:18.221648 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:18.223815 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: Bootstrap starting.
I20260928 22:48:18.224330 18472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:18.224547 18472 log.cc:824] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:18.225189 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: No bootstrap required, opened a new log
I20260928 22:48:18.226698 18472 raft_consensus.cc:348] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 22:48:18.226832 18472 raft_consensus.cc:374] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:18.226853 18472 raft_consensus.cc:729] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a664227856c745d69408b1fab9e4b6fe, State: Initialized, Role: FOLLOWER
I20260928 22:48:18.226958 18472 consensus_queue.cc:260] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 22:48:18.227028 18472 raft_consensus.cc:388] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:18.227057 18472 raft_consensus.cc:482] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:18.227088 18472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:18.227546 18472 raft_consensus.cc:504] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 22:48:18.227627 18472 leader_election.cc:302] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a664227856c745d69408b1fab9e4b6fe; no voters:
I20260928 22:48:18.227753 18472 leader_election.cc:288] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:18.227838 18475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:18.227977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18404
I20260928 22:48:18.227989 18472 sys_catalog.cc:564] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:18.228024 18475 raft_consensus.cc:686] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [term 1 LEADER]: Becoming Leader. State: Replica: a664227856c745d69408b1fab9e4b6fe, State: Running, Role: LEADER
I20260928 22:48:18.228086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:18.228109 18475 consensus_queue.cc:237] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 22:48:18.228278 392 external_mini_cluster.cc:1428] Setting key eca7704503bdbe249dc4b8cce9f7ea6d
I20260928 22:48:18.228415 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a664227856c745d69408b1fab9e4b6fe. Latest consensus state: current_term: 1 leader_uuid: "a664227856c745d69408b1fab9e4b6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 22:48:18.228479 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:18.228622 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a664227856c745d69408b1fab9e4b6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a664227856c745d69408b1fab9e4b6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 22:48:18.228682 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:18.228693 18482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:18.229269 18482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:18.230959 18482 catalog_manager.cc:1347] Generated new cluster ID: 6bfdcd309ba04f5cbf3d1fd8949fd2b7
I20260928 22:48:18.231012 18482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:18.237802 18482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:18.238513 18482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:18.238749 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.229108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.62:42083)
0928 22:48:18.229491 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.229500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.229531 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:48:18.229657 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.229663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.229721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.229854 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:18.230627 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.231618 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.232366 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.232848 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.232943 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.235124 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:18.235129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:18.235129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:18.235141 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:18.237179 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.237571 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.237574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.237574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.237693 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.237934 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.237935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.237935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.238159 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:18.238312 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.238393 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.238534 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":68,"threads_started":1}
I20260928 22:48:18.242956 18482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: Generated new TSK 0
I20260928 22:48:18.243130 18482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:48:18.269260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:18.349371 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:18.349572 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:18.349604 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:18.350760 18498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:18.350800 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:18.350835 18498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:18.352311 18498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:18.352538 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:18.352784 18498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:18.357582 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.357573 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.357581 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:18.357625 18498 server_base.cc:1034] running on GCE node
I20260928 22:48:18.357964 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:18.358197 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:18.359337 18498 hybrid_clock.cc:648] HybridClock initialized: now 1790635698359321 us; error 28 us; skew 500 ppm
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:18.361771 18498 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:48:18.362309 18498 webserver.cc:460] Webserver started at http://127.0.98.1:34861/ using document root <none> and password file <none>
I20260928 22:48:18.362502 18498 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:18.362548 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:18.362669 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:18.363729 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa7f82dff00747e1b214ac18e3cff42d"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "a57bc5cf6dc95a992719d2e482dc62d2"
server_key_iv: "4c93d53e1021a57e7852c3eabad549c1"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.364056 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa7f82dff00747e1b214ac18e3cff42d"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "a57bc5cf6dc95a992719d2e482dc62d2"
server_key_iv: "4c93d53e1021a57e7852c3eabad549c1"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.365350 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:18.366065 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.366240 18498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:18.366328 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "fa7f82dff00747e1b214ac18e3cff42d"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "a57bc5cf6dc95a992719d2e482dc62d2"
server_key_iv: "4c93d53e1021a57e7852c3eabad549c1"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.366387 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:18.385142 18498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:18.387044 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:18.387174 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:18.387380 18498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:18.387651 18498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:18.387682 18498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.387704 18498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:18.387728 18498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.393126 18498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33445
I20260928 22:48:18.393185 18626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33445 every 8 connection(s)
I20260928 22:48:18.393466 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:18.394065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18498
I20260928 22:48:18.394176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:18.394402 392 external_mini_cluster.cc:1428] Setting key 8f51efe547e370b30d33f8cea8f648f8
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 22:48:18.403656 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.394140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47607 (local address 127.0.98.62:42083)
0928 22:48:18.394193 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.394195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.394397 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:48:18.395584 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.395585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.395601 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.395630 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:18.396553 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.397331 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.398327 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.398638 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.398713 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.400757 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:18.400759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:18.400759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:18.400769 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:18.402273 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.402729 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.402730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.402731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.402794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.402993 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.402994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.402994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.403117 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:18.403167 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.403501 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.403584 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:48:18.403789 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.394087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:47607)
0928 22:48:18.394256 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.394306 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.395462 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.395822 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.395825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.396168 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:18.396452 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.396459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.397416 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.397417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.398236 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.398238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.398716 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.398717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.398824 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.399303 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:18.399311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:18.400633 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:18.402354 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.402363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.402365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.402641 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.402853 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.402854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.402854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.402949 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.403160 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:18.403163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:18.403341 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.403448 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.403585 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 22:48:18.404292 18627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 22:48:18.404414 18627 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:18.404567 18627 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 22:48:18.405088 18436 ts_manager.cc:194] Registered new tserver with Master: fa7f82dff00747e1b214ac18e3cff42d (127.0.98.1:33445)
I20260928 22:48:18.405704 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:48:18.419086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:18.497618 18632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:18.497824 18632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:18.497848 18632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:18.499012 18632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:18.499059 18632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:18.499085 18632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:18.500589 18632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:18.500834 18632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:18.501083 18632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:18.505888 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:18.506009 18632 server_base.cc:1034] running on GCE node
W20260928 22:48:18.505898 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.505900 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:18.506304 18632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:18.506539 18632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:18.507679 18632 hybrid_clock.cc:648] HybridClock initialized: now 1790635698507663 us; error 33 us; skew 500 ppm
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:18.514816 18632 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:48:18.515255 18632 webserver.cc:460] Webserver started at http://127.0.98.2:34487/ using document root <none> and password file <none>
I20260928 22:48:18.515479 18632 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:18.515528 18632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:18.515655 18632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:18.516681 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f4ebdadc2224328b9c00a5de5c9301e"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "518f76f773b9f8b6a87dd8b98dd78ab6"
server_key_iv: "df45184e46a8ad1598fdd5d1167f0eaf"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.517045 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f4ebdadc2224328b9c00a5de5c9301e"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "518f76f773b9f8b6a87dd8b98dd78ab6"
server_key_iv: "df45184e46a8ad1598fdd5d1167f0eaf"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.518250 18632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:18.518908 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.519052 18632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:18.519130 18632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "6f4ebdadc2224328b9c00a5de5c9301e"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "518f76f773b9f8b6a87dd8b98dd78ab6"
server_key_iv: "df45184e46a8ad1598fdd5d1167f0eaf"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.519196 18632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:18.529196 18632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:18.530925 18632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:18.531069 18632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:18.531289 18632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:18.531649 18632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:18.531693 18632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.531730 18632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:18.531747 18632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.537204 18632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33927
I20260928 22:48:18.537238 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33927 every 8 connection(s)
I20260928 22:48:18.537556 18632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:18.543589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18632
I20260928 22:48:18.543707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:18.543939 392 external_mini_cluster.cc:1428] Setting key 7ba55cdd5993d29c8257f293a7fda09c
I20260928 22:48:18.546759 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.538185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45841 (local address 127.0.98.62:42083)
0928 22:48:18.538262 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.538264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.538422 (+ 158us) server_negotiation.cc:407] Connection header received
0928 22:48:18.539366 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.539367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.539378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.539434 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:18.540057 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.540568 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.541686 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.541882 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.541940 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.544089 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:18.544090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:18.544091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:18.544098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:18.545451 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.545914 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.545914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.545915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.545979 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.546182 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.546182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.546183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.546316 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:18.546376 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.546599 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.546681 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:48:18.546916 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.538148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:45841)
0928 22:48:18.538305 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.538341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.539238 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.539500 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.539503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.539766 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:18.539958 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.539962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.540657 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.540658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.541612 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.541614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.541947 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.541948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.542051 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.542516 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:18.542525 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:18.543974 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:18.545531 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.545540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.545542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.545846 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.546037 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.546037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.546037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.546133 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.546367 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:18.546371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:18.546488 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.546566 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.546700 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 22:48:18.547451 18761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 22:48:18.547551 18761 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:18.547729 18761 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 22:48:18.548146 18436 ts_manager.cc:194] Registered new tserver with Master: 6f4ebdadc2224328b9c00a5de5c9301e (127.0.98.2:33927)
I20260928 22:48:18.548522 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45841
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:48:18.567942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:18.646646 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:18.646853 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:18.646874 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:18.647975 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:18.648013 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:18.648037 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:18.649483 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:18.649669 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:18.649847 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:18.654824 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.654843 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:18.654824 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:18.655037 18766 server_base.cc:1034] running on GCE node
I20260928 22:48:18.655321 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:18.655561 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:18.656703 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790635698656689 us; error 29 us; skew 500 ppm
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:18.659030 18766 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:48:18.659454 18766 webserver.cc:460] Webserver started at http://127.0.98.3:44981/ using document root <none> and password file <none>
I20260928 22:48:18.659650 18766 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:18.659698 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:18.659818 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:18.660929 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "71b70bfbb94942378b7ac6a168c87976"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "9a6027a808b2c39aa652d143ecd17b68"
server_key_iv: "9426c68037670e6b3f2b68d6eda76f8b"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.661260 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71b70bfbb94942378b7ac6a168c87976"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "9a6027a808b2c39aa652d143ecd17b68"
server_key_iv: "9426c68037670e6b3f2b68d6eda76f8b"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.662515 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:18.663197 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.663331 18766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.663410 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "71b70bfbb94942378b7ac6a168c87976"
format_stamp: "Formatted at 2026-09-28 22:48:18 on dist-test-slave-pxr1"
server_key: "9a6027a808b2c39aa652d143ecd17b68"
server_key_iv: "9426c68037670e6b3f2b68d6eda76f8b"
server_key_version: "encryptionkey@0"
I20260928 22:48:18.663470 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:18.677374 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:18.679569 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:18.679711 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:18.679937 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:18.680239 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:18.680281 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.680310 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:18.680334 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:18.685873 18766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44291
I20260928 22:48:18.685915 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44291 every 8 connection(s)
I20260928 22:48:18.686220 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:48:18 dist-test-slave-pxr1 krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:18.692607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18766
I20260928 22:48:18.692701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:18.692940 392 external_mini_cluster.cc:1428] Setting key b04a0d822298e9b08c78fb69c6fb5142
I20260928 22:48:18.695329 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.686945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60559 (local address 127.0.98.62:42083)
0928 22:48:18.687026 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.687028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.687165 (+ 137us) server_negotiation.cc:407] Connection header received
0928 22:48:18.688059 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.688060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.688072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.688122 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:18.688704 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.689229 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.690198 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.690395 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.690448 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.692815 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:18.692817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:18.692817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:18.692826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:18.694267 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.694618 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.694619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.694620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.694665 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.694844 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.694845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.694845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.694944 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:18.694988 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.695202 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.695259 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:48:18.695524 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.686904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:60559)
0928 22:48:18.687049 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.687086 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.687973 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.688179 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.688182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.688446 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:18.688641 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.688645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.689301 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.689302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.690110 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.690112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.690458 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.690459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.690556 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.691050 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:18.691058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:18.692678 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:18.694338 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.694344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.694346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.694563 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.694706 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:18.694707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:18.694707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:18.694802 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:18.694985 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:18.694988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:18.695103 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.695180 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.695308 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 22:48:18.695927 18895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 22:48:18.696025 18895 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:18.696214 18895 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 22:48:18.696581 18436 ts_manager.cc:194] Registered new tserver with Master: 71b70bfbb94942378b7ac6a168c87976 (127.0.98.3:44291)
I20260928 22:48:18.696985 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60559
I20260928 22:48:18.704572 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:18.711241 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.705882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:42083)
0928 22:48:18.705930 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.705932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.706004 (+ 72us) server_negotiation.cc:407] Connection header received
0928 22:48:18.706043 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.706043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.706051 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.706101 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:18.706619 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.707100 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.707784 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.707972 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.708030 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.709242 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:18.709243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:18.709243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:18.709249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:18.710300 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.710633 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.710634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.710634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.710684 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:18.710844 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:18.710844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:18.710845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:18.710971 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:18.711050 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.711131 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.711177 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:48:18.712685 18436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:18.714269 18436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:18.720621 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.0.98.2:33927)
0928 22:48:18.716815 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.716819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.716831 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:18.716953 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.716955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.716968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.717056 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:18.717329 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718477 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.719738 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720236 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720301 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720362 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.720505 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.720542 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":74,"threads_started":1}
I20260928 22:48:18.720618 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.0.1:32956)
0928 22:48:18.716747 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.716790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.716856 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.717160 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.717163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.717166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:18.717234 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.717236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718567 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.718568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.719667 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.719670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720323 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.720406 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720420 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.720483 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.720536 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260928 22:48:18.720788 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.0.98.1:33445)
0928 22:48:18.716770 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.716775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.716993 (+ 218us) server_negotiation.cc:407] Connection header received
0928 22:48:18.717056 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.717058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.717073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.717136 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:18.717329 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718440 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.719556 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720295 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720363 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720418 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.720631 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.720673 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260928 22:48:18.720815 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.98.3:44291)
0928 22:48:18.716909 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:18.716913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:18.717078 (+ 165us) server_negotiation.cc:407] Connection header received
0928 22:48:18.717192 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:18.717195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:18.717210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:18.717275 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:18.717488 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718198 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.719627 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720295 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720363 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720419 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:18.720672 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 22:48:18.720711 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260928 22:48:18.720875 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44291 (local address 127.0.0.1:39534)
0928 22:48:18.716968 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.716975 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.717105 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.717358 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.717358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.717360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:18.717408 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.717410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718291 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.718293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.719554 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.719557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720455 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.720536 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720545 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.720589 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.720633 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":47,"thread_start_us":60,"threads_started":1}
I20260928 22:48:18.721000 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:18.716381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33445 (local address 127.0.0.1:34110)
0928 22:48:18.716924 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:18.716932 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:18.716989 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:18.717217 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:18.717217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:18.717219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:18.717253 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.717254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.718558 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.718559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.719481 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:18.719483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:18.720454 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:18.720455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:18.720536 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:18.720543 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:48:18.720894 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 22:48:18.720940 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":47,"thread_start_us":84,"threads_started":1}
I20260928 22:48:18.721705 18561 tablet_service.cc:1467] Processing CreateTablet for tablet 904a2ec8bd634525b516836bd3f2a6af (DEFAULT_TABLE table=test-table [id=87097b8139dc425ba5da42973be1665c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:18.721683 18829 tablet_service.cc:1467] Processing CreateTablet for tablet 904a2ec8bd634525b516836bd3f2a6af (DEFAULT_TABLE table=test-table [id=87097b8139dc425ba5da42973be1665c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:18.721925 18695 tablet_service.cc:1467] Processing CreateTablet for tablet 904a2ec8bd634525b516836bd3f2a6af (DEFAULT_TABLE table=test-table [id=87097b8139dc425ba5da42973be1665c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:18.722052 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a2ec8bd634525b516836bd3f2a6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:18.722052 18829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a2ec8bd634525b516836bd3f2a6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:18.722240 18695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904a2ec8bd634525b516836bd3f2a6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:18.724809 18909 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Bootstrap starting.
I20260928 22:48:18.725211 18911 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Bootstrap starting.
I20260928 22:48:18.725643 18909 tablet_bootstrap.cc:654] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:18.725849 18911 tablet_bootstrap.cc:654] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:18.726249 18909 log.cc:824] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:18.726373 18911 log.cc:824] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:18.726665 18910 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Bootstrap starting.
I20260928 22:48:18.726953 18909 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: No bootstrap required, opened a new log
I20260928 22:48:18.727013 18909 ts_tablet_manager.cc:1397] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:18.727185 18910 tablet_bootstrap.cc:654] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:18.727651 18910 log.cc:824] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:18.728178 18911 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: No bootstrap required, opened a new log
I20260928 22:48:18.728250 18911 ts_tablet_manager.cc:1397] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:48:18.728530 18910 tablet_bootstrap.cc:492] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: No bootstrap required, opened a new log
I20260928 22:48:18.728600 18910 ts_tablet_manager.cc:1397] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:18.728838 18909 raft_consensus.cc:348] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.729023 18909 raft_consensus.cc:374] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:18.729065 18909 raft_consensus.cc:729] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4ebdadc2224328b9c00a5de5c9301e, State: Initialized, Role: FOLLOWER
I20260928 22:48:18.729192 18909 consensus_queue.cc:260] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.729508 18909 ts_tablet_manager.cc:1428] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:18.729640 18761 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 22:48:18.730438 18911 raft_consensus.cc:348] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.730575 18910 raft_consensus.cc:348] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.730728 18911 raft_consensus.cc:374] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:18.730728 18910 raft_consensus.cc:374] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:18.730775 18910 raft_consensus.cc:729] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7f82dff00747e1b214ac18e3cff42d, State: Initialized, Role: FOLLOWER
I20260928 22:48:18.730777 18911 raft_consensus.cc:729] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b70bfbb94942378b7ac6a168c87976, State: Initialized, Role: FOLLOWER
I20260928 22:48:18.730883 18910 consensus_queue.cc:260] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.730888 18911 consensus_queue.cc:260] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:18.731129 18627 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 22:48:18.731134 18910 ts_tablet_manager.cc:1428] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:18.731189 18895 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 22:48:18.731446 18911 ts_tablet_manager.cc:1428] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 22:48:18.788152 18762 tablet.cc:2369] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:18.894191 18628 tablet.cc:2369] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:18.936789 18896 tablet.cc:2369] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:19.005764 18917 raft_consensus.cc:482] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:19.005908 18917 raft_consensus.cc:504] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:19.006305 18917 leader_election.cc:288] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7f82dff00747e1b214ac18e3cff42d (127.0.98.1:33445), 6f4ebdadc2224328b9c00a5de5c9301e (127.0.98.2:33927)
I20260928 22:48:19.009111 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.006540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52897 (local address 127.0.98.1:33445)
0928 22:48:19.006604 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.006606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.006615 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:19.006686 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.006687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.006701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.006751 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.006907 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.007431 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008386 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.008787 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008840 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.008850 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.008997 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.009036 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:48:19.009128 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.006552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59611 (local address 127.0.98.2:33927)
0928 22:48:19.006597 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.006599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.006770 (+ 171us) server_negotiation.cc:407] Connection header received
0928 22:48:19.006823 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.006824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.006837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.006889 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.007032 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.007490 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008413 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.008815 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008884 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.008894 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.009017 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.009054 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:48:19.009128 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.006486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33445 (local address 127.0.98.3:52897)
0928 22:48:19.006531 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.006543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.006610 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.006773 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.006773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.006776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.006847 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.006849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.007500 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.007501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.008296 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.008299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.008854 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.008931 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.008934 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.008978 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.009018 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":22}
I20260928 22:48:19.009135 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.006486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.98.3:59611)
0928 22:48:19.006690 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.006701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.006759 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.006892 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.006893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.006894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.006972 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.006974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.007570 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.007571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.008325 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.008326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.008895 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.008896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.008951 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.008954 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.009005 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.009043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":51,"thread_start_us":96,"threads_started":1}
I20260928 22:48:19.009510 18715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a2ec8bd634525b516836bd3f2a6af" candidate_uuid: "71b70bfbb94942378b7ac6a168c87976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" is_pre_election: true
I20260928 22:48:19.009544 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a2ec8bd634525b516836bd3f2a6af" candidate_uuid: "71b70bfbb94942378b7ac6a168c87976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7f82dff00747e1b214ac18e3cff42d" is_pre_election: true
I20260928 22:48:19.009649 18715 raft_consensus.cc:2454] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71b70bfbb94942378b7ac6a168c87976 in term 0.
I20260928 22:48:19.009694 18581 raft_consensus.cc:2454] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71b70bfbb94942378b7ac6a168c87976 in term 0.
I20260928 22:48:19.009881 18782 leader_election.cc:302] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4ebdadc2224328b9c00a5de5c9301e, 71b70bfbb94942378b7ac6a168c87976; no voters:
I20260928 22:48:19.010046 18917 raft_consensus.cc:2781] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:19.010102 18917 raft_consensus.cc:482] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:19.010125 18917 raft_consensus.cc:3037] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.010795 18917 raft_consensus.cc:504] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:19.010922 18917 leader_election.cc:288] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [CANDIDATE]: Term 1 election: Requested vote from peers fa7f82dff00747e1b214ac18e3cff42d (127.0.98.1:33445), 6f4ebdadc2224328b9c00a5de5c9301e (127.0.98.2:33927)
I20260928 22:48:19.011101 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a2ec8bd634525b516836bd3f2a6af" candidate_uuid: "71b70bfbb94942378b7ac6a168c87976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7f82dff00747e1b214ac18e3cff42d"
I20260928 22:48:19.011096 18715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904a2ec8bd634525b516836bd3f2a6af" candidate_uuid: "71b70bfbb94942378b7ac6a168c87976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4ebdadc2224328b9c00a5de5c9301e"
I20260928 22:48:19.011169 18715 raft_consensus.cc:3037] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.011171 18581 raft_consensus.cc:3037] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.011837 18581 raft_consensus.cc:2454] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71b70bfbb94942378b7ac6a168c87976 in term 1.
I20260928 22:48:19.011900 18715 raft_consensus.cc:2454] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71b70bfbb94942378b7ac6a168c87976 in term 1.
I20260928 22:48:19.012041 18784 leader_election.cc:302] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f4ebdadc2224328b9c00a5de5c9301e, 71b70bfbb94942378b7ac6a168c87976, fa7f82dff00747e1b214ac18e3cff42d; no voters:
I20260928 22:48:19.012142 18917 raft_consensus.cc:2781] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:19.012271 18917 raft_consensus.cc:686] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 1 LEADER]: Becoming Leader. State: Replica: 71b70bfbb94942378b7ac6a168c87976, State: Running, Role: LEADER
I20260928 22:48:19.012374 18917 consensus_queue.cc:237] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } }
I20260928 22:48:19.013222 18436 catalog_manager.cc:5495] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71b70bfbb94942378b7ac6a168c87976 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "71b70bfbb94942378b7ac6a168c87976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71b70bfbb94942378b7ac6a168c87976" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44291 } health_report { overall_health: HEALTHY } } }
I20260928 22:48:19.077632 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.075263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.98.3:44291)
0928 22:48:19.075337 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.075339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.075348 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:19.075393 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.075394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.075413 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.075472 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:19.075620 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.076088 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.076890 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.077075 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.077137 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.077286 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:19.077380 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:19.077467 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.077522 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.077531 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:48:19.079406 18715 raft_consensus.cc:1261] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 1 FOLLOWER]: Refusing update from remote peer 71b70bfbb94942378b7ac6a168c87976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:19.079423 18581 raft_consensus.cc:1261] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 1 FOLLOWER]: Refusing update from remote peer 71b70bfbb94942378b7ac6a168c87976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:19.079768 18919 consensus_queue.cc:1035] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7f82dff00747e1b214ac18e3cff42d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:19.079900 18917 consensus_queue.cc:1035] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4ebdadc2224328b9c00a5de5c9301e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:19.082545 18924 mvcc.cc:205] Tried to move back new op lower bound from 7334443823426818048 to 7334443823155326976. Current Snapshot: MvccSnapshot[applied={T|T < 7334443823426818048 or (T in {7334443823426818048})}]
I20260928 22:48:19.082777 18923 mvcc.cc:205] Tried to move back new op lower bound from 7334443823426818048 to 7334443823155326976. Current Snapshot: MvccSnapshot[applied={T|T < 7334443823426818048}]
I20260928 22:48:19.084206 18435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:19.084306 18435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:19.085166 18435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a664227856c745d69408b1fab9e4b6fe: Sending DeleteTablet for 3 replicas of tablet 904a2ec8bd634525b516836bd3f2a6af
I20260928 22:48:19.085505 18695 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a2ec8bd634525b516836bd3f2a6af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32956
I20260928 22:48:19.085497 18561 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a2ec8bd634525b516836bd3f2a6af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34110
I20260928 22:48:19.085716 18932 tablet_replica.cc:330] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d: stopping tablet replica
I20260928 22:48:19.085812 18932 raft_consensus.cc:2229] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:19.085831 18829 tablet_service.cc:1514] Processing DeleteTablet for tablet 904a2ec8bd634525b516836bd3f2a6af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39534
I20260928 22:48:19.085933 18932 raft_consensus.cc:2258] T 904a2ec8bd634525b516836bd3f2a6af P fa7f82dff00747e1b214ac18e3cff42d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:19.086102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18498
I20260928 22:48:19.091866 18931 tablet_replica.cc:330] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e: stopping tablet replica
I20260928 22:48:19.092003 18931 raft_consensus.cc:2229] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:19.092135 18931 raft_consensus.cc:2258] T 904a2ec8bd634525b516836bd3f2a6af P 6f4ebdadc2224328b9c00a5de5c9301e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:19.092296 18933 tablet_replica.cc:330] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: stopping tablet replica
I20260928 22:48:19.092394 18933 raft_consensus.cc:2229] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:19.092496 18933 raft_consensus.cc:2258] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:48:19.092629 18422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33445 (error 108)
I20260928 22:48:19.092880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18632
I20260928 22:48:19.093097 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.092953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33445 (local address 127.0.0.1:34118)
0928 22:48:19.093022 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.093053 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 22:48:19.093694 18422 catalog_manager.cc:4584] TS fa7f82dff00747e1b214ac18e3cff42d (127.0.98.1:33445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 904a2ec8bd634525b516836bd3f2a6af: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33445: connect: Connection refused (error 111)
I20260928 22:48:19.093731 18933 ts_tablet_manager.cc:1905] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:19.095146 18933 ts_tablet_manager.cc:1918] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:19.095266 18933 log.cc:1196] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/904a2ec8bd634525b516836bd3f2a6af
I20260928 22:48:19.095604 18933 ts_tablet_manager.cc:1939] T 904a2ec8bd634525b516836bd3f2a6af P 71b70bfbb94942378b7ac6a168c87976: Deleting consensus metadata
I20260928 22:48:19.096088 18420 catalog_manager.cc:4841] TS 71b70bfbb94942378b7ac6a168c87976 (127.0.98.3:44291): tablet 904a2ec8bd634525b516836bd3f2a6af (table test-table [id=87097b8139dc425ba5da42973be1665c]) successfully deleted
I20260928 22:48:19.098579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18766
I20260928 22:48:19.098728 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.098595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.0.1:32962)
0928 22:48:19.098664 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.098696 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 22:48:19.102926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18404
2026-09-28T22:48:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2080 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T22:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:19Z Disabled control of system clock
I20260928 22:48:19.129145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40039
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:19.210353 18938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:19.210584 18938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:19.210605 18938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:19.211798 18938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:19.211846 18938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:19.211858 18938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:19.211869 18938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:19.211879 18938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:19.213316 18938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:19.213524 18938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:19.213814 18938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:19.218698 18946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.218686 18944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.218688 18943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:19.218968 18938 server_base.cc:1034] running on GCE node
I20260928 22:48:19.219226 18938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:19.219605 18938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:19.220805 18938 hybrid_clock.cc:648] HybridClock initialized: now 1790635699220785 us; error 39 us; skew 500 ppm
I20260928 22:48:19.222055 18938 webserver.cc:460] Webserver started at http://127.0.98.62:40581/ using document root <none> and password file <none>
I20260928 22:48:19.222287 18938 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:19.222337 18938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:19.222456 18938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:19.223505 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "41f2ec221ca74dd7a0b394876c359001"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "cbc008aecd13d58b24a0b463f43a91cc"
server_key_iv: "6d51dea290ed78efe9583da1bbc435e9"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.223843 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "41f2ec221ca74dd7a0b394876c359001"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "cbc008aecd13d58b24a0b463f43a91cc"
server_key_iv: "6d51dea290ed78efe9583da1bbc435e9"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.225060 18938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:19.225814 18952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.225980 18938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:19.226071 18938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "41f2ec221ca74dd7a0b394876c359001"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "cbc008aecd13d58b24a0b463f43a91cc"
server_key_iv: "6d51dea290ed78efe9583da1bbc435e9"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.226130 18938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:19.263746 18938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:19.264073 18938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:19.264206 18938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:19.267927 18938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40039
I20260928 22:48:19.267964 19004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40039 every 8 connection(s)
I20260928 22:48:19.268287 18938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:19.269271 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:19.271541 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: Bootstrap starting.
I20260928 22:48:19.272089 19005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:19.272343 19005 log.cc:824] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:19.273047 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: No bootstrap required, opened a new log
I20260928 22:48:19.273653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 18938
I20260928 22:48:19.273747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:19.273916 392 external_mini_cluster.cc:1428] Setting key e1ea2284e739ffa10e8a9e49de10bbe6
I20260928 22:48:19.275101 19005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } }
I20260928 22:48:19.275288 19005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:19.275324 19005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f2ec221ca74dd7a0b394876c359001, State: Initialized, Role: FOLLOWER
I20260928 22:48:19.275449 19005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } }
I20260928 22:48:19.275532 19005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:19.275560 19005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:19.275604 19005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.276147 19005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } }
I20260928 22:48:19.276243 19005 leader_election.cc:302] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f2ec221ca74dd7a0b394876c359001; no voters:
I20260928 22:48:19.276422 19005 leader_election.cc:288] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:19.276453 19010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:19.276551 19010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [term 1 LEADER]: Becoming Leader. State: Replica: 41f2ec221ca74dd7a0b394876c359001, State: Running, Role: LEADER
I20260928 22:48:19.276633 19010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } }
I20260928 22:48:19.276684 19005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:19.277004 19010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f2ec221ca74dd7a0b394876c359001. Latest consensus state: current_term: 1 leader_uuid: "41f2ec221ca74dd7a0b394876c359001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } }
I20260928 22:48:19.277084 19010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:19.277014 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f2ec221ca74dd7a0b394876c359001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f2ec221ca74dd7a0b394876c359001" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } }
I20260928 22:48:19.277127 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:19.277307 19016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:19.277813 19016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:19.279371 19016 catalog_manager.cc:1347] Generated new cluster ID: 8890b8c178fb405b81cf14ece836d08e
I20260928 22:48:19.279424 19016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:19.280141 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.274404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.0.98.62:40039)
0928 22:48:19.274631 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.274640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.274676 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:48:19.275067 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.275076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.275154 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.275310 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:19.275513 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.277024 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.278232 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.278780 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.278903 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.279530 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:19.279534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:19.279534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:19.279703 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:19.279735 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:19.279745 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:19.279799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.279887 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.279965 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260928 22:48:19.303073 19016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:19.303638 19016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:19.305908 19016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: Generated new TSK 0
I20260928 22:48:19.306071 19016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:19.310487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:19.404652 19029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:19.404883 19029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:19.404914 19029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:19.406095 19029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:19.406133 19029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:19.406167 19029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:19.407472 19029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:19.407692 19029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:19.407961 19029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:19.412818 19034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.412917 19035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.412926 19037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:19.413447 19029 server_base.cc:1034] running on GCE node
I20260928 22:48:19.413666 19029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:19.413924 19029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:19.415086 19029 hybrid_clock.cc:648] HybridClock initialized: now 1790635699415065 us; error 35 us; skew 500 ppm
I20260928 22:48:19.416520 19029 webserver.cc:460] Webserver started at http://127.0.98.1:46613/ using document root <none> and password file <none>
I20260928 22:48:19.416818 19029 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:19.416883 19029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:19.417011 19029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:19.418408 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "da5342cd97834d37a2e908d6212a9d14"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "c1e1c1fd52f7c4227298bc20d1201cec"
server_key_iv: "1d1e0c8d5ae26328c973e54898888433"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.418834 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da5342cd97834d37a2e908d6212a9d14"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "c1e1c1fd52f7c4227298bc20d1201cec"
server_key_iv: "1d1e0c8d5ae26328c973e54898888433"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.420504 19029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:19.421366 19043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.421520 19029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:19.421607 19029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "da5342cd97834d37a2e908d6212a9d14"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "c1e1c1fd52f7c4227298bc20d1201cec"
server_key_iv: "1d1e0c8d5ae26328c973e54898888433"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.421676 19029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:19.442164 19029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:19.442447 19029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:19.442544 19029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:19.442737 19029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:19.443010 19029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:19.443050 19029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.443076 19029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:19.443090 19029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.448342 19029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34711
I20260928 22:48:19.448398 19156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34711 every 8 connection(s)
I20260928 22:48:19.448700 19029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:19.453539 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.449428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48169 (local address 127.0.98.62:40039)
0928 22:48:19.449511 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.449513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.449710 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:48:19.450180 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.450181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.450192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.450245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:19.450865 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.451354 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.452335 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.452534 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.452599 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.453332 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:19.453334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:19.453334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:19.453338 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:19.453352 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:19.453356 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:19.453396 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.453450 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.453479 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:48:19.453734 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.449396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.98.1:48169)
0928 22:48:19.449584 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.449625 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.450107 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.450305 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.450308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.450317 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:19.450802 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.450806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.451425 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.451426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.452231 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.452234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.452607 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.452608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.452710 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.453211 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:19.453212 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:19.453239 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:19.453261 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:19.453402 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:19.453405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:19.453407 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.453451 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.453527 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260928 22:48:19.454151 19157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40039
I20260928 22:48:19.454232 19157 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:19.454397 19157 heartbeater.cc:511] Master 127.0.98.62:40039 requested a full tablet report, sending...
I20260928 22:48:19.454842 18969 ts_manager.cc:194] Registered new tserver with Master: da5342cd97834d37a2e908d6212a9d14 (127.0.98.1:34711)
I20260928 22:48:19.455436 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48169
I20260928 22:48:19.455866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19029
I20260928 22:48:19.455977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:19.456174 392 external_mini_cluster.cc:1428] Setting key ebcbebd778ddee0858b2960afb0a36c6
I20260928 22:48:19.456970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:19.535883 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:19.536106 19160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:19.536129 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:19.537293 19160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:19.537329 19160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:19.537354 19160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:19.538681 19160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:19.538885 19160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:19.539145 19160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:19.543911 19165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.543911 19166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:19.544083 19160 server_base.cc:1034] running on GCE node
W20260928 22:48:19.543905 19168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:19.544330 19160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:19.544538 19160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:19.545670 19160 hybrid_clock.cc:648] HybridClock initialized: now 1790635699545648 us; error 38 us; skew 500 ppm
I20260928 22:48:19.546900 19160 webserver.cc:460] Webserver started at http://127.0.98.2:38873/ using document root <none> and password file <none>
I20260928 22:48:19.547120 19160 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:19.547165 19160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:19.547291 19160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:19.548333 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "27d5f1287cff41799f23826f352dfa13"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "7bc0f6a416bee41f550ea33056e2fad0"
server_key_iv: "02956296722bcbd0e7bb625ef57d591e"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.548647 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27d5f1287cff41799f23826f352dfa13"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "7bc0f6a416bee41f550ea33056e2fad0"
server_key_iv: "02956296722bcbd0e7bb625ef57d591e"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.549865 19160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:19.550545 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.550690 19160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:19.550772 19160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "27d5f1287cff41799f23826f352dfa13"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "7bc0f6a416bee41f550ea33056e2fad0"
server_key_iv: "02956296722bcbd0e7bb625ef57d591e"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.550827 19160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:19.583395 19160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:19.583676 19160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:19.583777 19160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:19.583967 19160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:19.584301 19160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:19.584334 19160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.584357 19160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:19.584379 19160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.589830 19160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45031
I20260928 22:48:19.589882 19287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45031 every 8 connection(s)
I20260928 22:48:19.590201 19160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:19.591485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19160
I20260928 22:48:19.591583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:19.591768 392 external_mini_cluster.cc:1428] Setting key 51eadc8e3c94ce357f24891a7cc8d0fa
I20260928 22:48:19.592768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--builtin_ntp_servers=127.0.98.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:48:19.595923 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.590873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45435 (local address 127.0.98.62:40039)
0928 22:48:19.590933 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.590934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.591138 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:48:19.591705 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.591706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.591721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.591795 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:19.592624 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.593333 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.594598 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.594790 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.594852 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.595688 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:19.595690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:19.595690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:19.595696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:19.595711 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:19.595715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:19.595750 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.595837 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.595863 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:48:19.596174 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.590833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.98.2:45435)
0928 22:48:19.591026 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.591058 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.591569 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.591869 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.591873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.591884 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:19.592541 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.592546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.593415 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.593416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.594495 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.594499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.594865 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.594866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.595014 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.595534 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:19.595535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:19.595573 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:19.595602 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:19.595764 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:19.595768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:19.595772 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.595838 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.595948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260928 22:48:19.596706 19288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40039
I20260928 22:48:19.596824 19288 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:19.596997 19288 heartbeater.cc:511] Master 127.0.98.62:40039 requested a full tablet report, sending...
I20260928 22:48:19.597364 18969 ts_manager.cc:194] Registered new tserver with Master: 27d5f1287cff41799f23826f352dfa13 (127.0.98.2:45031)
I20260928 22:48:19.597870 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45435
W20260928 22:48:19.672993 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:19.673207 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:19.673230 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:19.674396 19291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:19.674429 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:19.674453 19291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:19.675758 19291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:19.675976 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:19.676257 19291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:19.681216 19296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.681207 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:19.681207 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:19.681316 19291 server_base.cc:1034] running on GCE node
I20260928 22:48:19.681638 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:19.681893 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:19.683058 19291 hybrid_clock.cc:648] HybridClock initialized: now 1790635699683045 us; error 27 us; skew 500 ppm
I20260928 22:48:19.684322 19291 webserver.cc:460] Webserver started at http://127.0.98.3:38841/ using document root <none> and password file <none>
I20260928 22:48:19.684563 19291 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:19.684612 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:19.684764 19291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:19.685822 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "23a99f5734e44abfb1c6bc3cd6706219"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "9a0dabc866957e960234e9917425c168"
server_key_iv: "1cfa921097f610efde34f259cfb01355"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.686149 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23a99f5734e44abfb1c6bc3cd6706219"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "9a0dabc866957e960234e9917425c168"
server_key_iv: "1cfa921097f610efde34f259cfb01355"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.687428 19291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:48:19.688208 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.688412 19291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:19.688496 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "23a99f5734e44abfb1c6bc3cd6706219"
format_stamp: "Formatted at 2026-09-28 22:48:19 on dist-test-slave-pxr1"
server_key: "9a0dabc866957e960234e9917425c168"
server_key_iv: "1cfa921097f610efde34f259cfb01355"
server_key_version: "encryptionkey@0"
I20260928 22:48:19.688570 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:19.695668 19291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:19.695948 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:19.696084 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:19.696285 19291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:19.696595 19291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:19.696630 19291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.696660 19291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:19.696677 19291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:19.702180 19291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37461
I20260928 22:48:19.702245 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37461 every 8 connection(s)
I20260928 22:48:19.702548 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:19.707548 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.703232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53415 (local address 127.0.98.62:40039)
0928 22:48:19.703274 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.703276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.703506 (+ 230us) server_negotiation.cc:407] Connection header received
0928 22:48:19.704019 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.704020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.704032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.704085 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:19.704778 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.705277 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.706246 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.706441 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.706494 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.707289 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:19.707290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:19.707291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:19.707296 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:19.707311 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:19.707315 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:19.707356 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.707429 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.707464 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:19.707818 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.703203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.98.3:53415)
0928 22:48:19.703389 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.703423 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.703914 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.704161 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.704166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.704175 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:19.704667 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.704672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.705361 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.705362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.706162 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.706165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.706520 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.706521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.706647 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.707152 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:19.707152 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:19.707181 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:19.707206 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:19.707362 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:19.707367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:19.707370 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.707429 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.707571 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 22:48:19.707844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19291
I20260928 22:48:19.707955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:19.708149 392 external_mini_cluster.cc:1428] Setting key b02781e24cbf54bc281ec3bb5e0feb42
I20260928 22:48:19.708292 19419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40039
I20260928 22:48:19.708379 19419 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:19.708555 19419 heartbeater.cc:511] Master 127.0.98.62:40039 requested a full tablet report, sending...
I20260928 22:48:19.708940 18969 ts_manager.cc:194] Registered new tserver with Master: 23a99f5734e44abfb1c6bc3cd6706219 (127.0.98.3:37461)
I20260928 22:48:19.709313 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53415
I20260928 22:48:19.719908 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:19.724192 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.721300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.0.98.62:40039)
0928 22:48:19.721343 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.721345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.721463 (+ 118us) server_negotiation.cc:407] Connection header received
0928 22:48:19.721528 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.721529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.721542 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.721591 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:19.721751 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.722250 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.722989 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.723184 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.723239 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.723914 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:19.723915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:19.723916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:19.723920 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:19.723936 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:19.723941 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:19.724020 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.724091 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.724102 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:19.725622 18969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:19.727203 18969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:19.733129 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.3:37461)
0928 22:48:19.729572 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.729577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.729766 (+ 189us) server_negotiation.cc:407] Connection header received
0928 22:48:19.729907 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.729910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.729926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.729994 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.730168 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.731211 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732203 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.732767 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732831 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.732919 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.733001 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.733045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 22:48:19.733119 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37461 (local address 127.0.0.1:33712)
0928 22:48:19.729701 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.729711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.729795 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.730061 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.730062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.730064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.730097 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.730099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.731284 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.731285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.732139 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.732141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.732830 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.732905 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.732918 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.732987 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.733040 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":91,"threads_started":1}
I20260928 22:48:19.733490 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.98.2:45031)
0928 22:48:19.729478 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.729483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.729572 (+ 89us) server_negotiation.cc:407] Connection header received
0928 22:48:19.729668 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.729671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.729687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.729756 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.729951 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.730785 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732121 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.732876 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732943 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.733025 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.733368 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.733408 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 22:48:19.733482 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34711 (local address 127.0.0.1:52232)
0928 22:48:19.729316 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.729351 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.729425 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.729887 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.729887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.729889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.729932 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.729934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.731200 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.731201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.732234 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.732236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.733167 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.733168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.733333 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.733337 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.733377 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.733417 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260928 22:48:19.733692 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45031 (local address 127.0.0.1:39082)
0928 22:48:19.729512 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.729519 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.729599 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.729766 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.729769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.729771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.729863 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.729865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.730861 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.730861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.732039 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.732042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.733059 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.733060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.733117 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.733121 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.733591 (+ 470us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.733632 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260928 22:48:19.733757 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.729521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.98.1:34711)
0928 22:48:19.729666 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.729669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.729687 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:48:19.729731 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.729733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.729747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.729819 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.730003 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.731112 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.732293 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.733009 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.733064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.733138 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.733640 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.733686 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260928 22:48:19.733804 19353 tablet_service.cc:1467] Processing CreateTablet for tablet 73d90342cb284a6294d8592e1eddcd1f (DEFAULT_TABLE table=test-table [id=ccc52ec3b2e74cc89bd13e0985a017ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:19.734179 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d90342cb284a6294d8592e1eddcd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:19.734339 19091 tablet_service.cc:1467] Processing CreateTablet for tablet 73d90342cb284a6294d8592e1eddcd1f (DEFAULT_TABLE table=test-table [id=ccc52ec3b2e74cc89bd13e0985a017ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:19.734640 19091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d90342cb284a6294d8592e1eddcd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:19.737162 19222 tablet_service.cc:1467] Processing CreateTablet for tablet 73d90342cb284a6294d8592e1eddcd1f (DEFAULT_TABLE table=test-table [id=ccc52ec3b2e74cc89bd13e0985a017ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:19.737458 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d90342cb284a6294d8592e1eddcd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:19.737469 19433 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Bootstrap starting.
I20260928 22:48:19.737775 19434 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: Bootstrap starting.
I20260928 22:48:19.738107 19433 tablet_bootstrap.cc:654] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:19.738258 19434 tablet_bootstrap.cc:654] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:19.738415 19433 log.cc:824] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:19.738456 19434 log.cc:824] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:19.739110 19433 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: No bootstrap required, opened a new log
I20260928 22:48:19.739117 19434 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: No bootstrap required, opened a new log
I20260928 22:48:19.739169 19433 ts_tablet_manager.cc:1397] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:19.739169 19434 ts_tablet_manager.cc:1397] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:19.740504 19437 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Bootstrap starting.
I20260928 22:48:19.741099 19437 tablet_bootstrap.cc:654] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:19.741356 19437 log.cc:824] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:19.741628 19433 raft_consensus.cc:348] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.741804 19433 raft_consensus.cc:374] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:19.741850 19433 raft_consensus.cc:729] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a99f5734e44abfb1c6bc3cd6706219, State: Initialized, Role: FOLLOWER
I20260928 22:48:19.741809 19434 raft_consensus.cc:348] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.741972 19434 raft_consensus.cc:374] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:19.741966 19433 consensus_queue.cc:260] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.742017 19434 raft_consensus.cc:729] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5342cd97834d37a2e908d6212a9d14, State: Initialized, Role: FOLLOWER
I20260928 22:48:19.742034 19437 tablet_bootstrap.cc:492] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: No bootstrap required, opened a new log
I20260928 22:48:19.742092 19437 ts_tablet_manager.cc:1397] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:19.742154 19434 consensus_queue.cc:260] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.742226 19433 ts_tablet_manager.cc:1428] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:48:19.742342 19419 heartbeater.cc:503] Master 127.0.98.62:40039 was elected leader, sending a full tablet report...
I20260928 22:48:19.742386 19434 ts_tablet_manager.cc:1428] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:48:19.742504 19157 heartbeater.cc:503] Master 127.0.98.62:40039 was elected leader, sending a full tablet report...
I20260928 22:48:19.744318 19437 raft_consensus.cc:348] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.744444 19437 raft_consensus.cc:374] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:19.744464 19437 raft_consensus.cc:729] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d5f1287cff41799f23826f352dfa13, State: Initialized, Role: FOLLOWER
I20260928 22:48:19.744570 19437 consensus_queue.cc:260] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.744837 19437 ts_tablet_manager.cc:1428] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 22:48:19.744930 19288 heartbeater.cc:503] Master 127.0.98.62:40039 was elected leader, sending a full tablet report...
I20260928 22:48:19.839643 19440 raft_consensus.cc:482] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:19.839784 19440 raft_consensus.cc:504] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.840090 19440 leader_election.cc:288] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27d5f1287cff41799f23826f352dfa13 (127.0.98.2:45031), 23a99f5734e44abfb1c6bc3cd6706219 (127.0.98.3:37461)
W20260928 22:48:19.840683 19289 tablet.cc:2369] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:19.842991 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.840352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52649 (local address 127.0.98.2:45031)
0928 22:48:19.840423 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.840426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.840439 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:48:19.840474 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.840476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.840486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.840534 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.840710 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.841249 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842132 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.842678 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842752 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.842762 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.842894 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.842927 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:48:19.843001 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.840267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45031 (local address 127.0.98.1:52649)
0928 22:48:19.840325 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.840339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.840404 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.840555 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.840556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.840557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.840632 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.840634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.841326 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.841327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.842056 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.842058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.842765 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.842827 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.842831 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.842882 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.842926 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:48:19.843055 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.840411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34089 (local address 127.0.98.3:37461)
0928 22:48:19.840453 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.840455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.840520 (+ 65us) server_negotiation.cc:407] Connection header received
0928 22:48:19.840591 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.840592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.840600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.840655 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:19.840865 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.841343 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842218 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.842759 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842816 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.842823 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.842961 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.842998 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:48:19.843243 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.840277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37461 (local address 127.0.98.1:34089)
0928 22:48:19.840454 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.840462 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:19.840523 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:19.840667 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:19.840668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:19.840670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:19.840770 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.840772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.841417 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.841418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.842146 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:19.842147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.842834 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.842835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:19.842898 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.842901 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:19.842948 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 22:48:19.842979 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260928 22:48:19.843369 19242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d90342cb284a6294d8592e1eddcd1f" candidate_uuid: "da5342cd97834d37a2e908d6212a9d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d5f1287cff41799f23826f352dfa13" is_pre_election: true
I20260928 22:48:19.843514 19242 raft_consensus.cc:2454] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5342cd97834d37a2e908d6212a9d14 in term 0.
I20260928 22:48:19.843557 19373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d90342cb284a6294d8592e1eddcd1f" candidate_uuid: "da5342cd97834d37a2e908d6212a9d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a99f5734e44abfb1c6bc3cd6706219" is_pre_election: true
I20260928 22:48:19.843706 19373 raft_consensus.cc:2454] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5342cd97834d37a2e908d6212a9d14 in term 0.
I20260928 22:48:19.843781 19046 leader_election.cc:302] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d5f1287cff41799f23826f352dfa13, da5342cd97834d37a2e908d6212a9d14; no voters:
I20260928 22:48:19.843902 19440 raft_consensus.cc:2781] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:19.843997 19440 raft_consensus.cc:482] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:19.844049 19440 raft_consensus.cc:3037] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.844769 19440 raft_consensus.cc:504] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.844897 19440 leader_election.cc:288] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [CANDIDATE]: Term 1 election: Requested vote from peers 27d5f1287cff41799f23826f352dfa13 (127.0.98.2:45031), 23a99f5734e44abfb1c6bc3cd6706219 (127.0.98.3:37461)
I20260928 22:48:19.845039 19373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d90342cb284a6294d8592e1eddcd1f" candidate_uuid: "da5342cd97834d37a2e908d6212a9d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a99f5734e44abfb1c6bc3cd6706219"
I20260928 22:48:19.845103 19373 raft_consensus.cc:3037] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.845770 19373 raft_consensus.cc:2454] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5342cd97834d37a2e908d6212a9d14 in term 1.
I20260928 22:48:19.845870 19242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d90342cb284a6294d8592e1eddcd1f" candidate_uuid: "da5342cd97834d37a2e908d6212a9d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d5f1287cff41799f23826f352dfa13"
I20260928 22:48:19.845932 19047 leader_election.cc:302] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a99f5734e44abfb1c6bc3cd6706219, da5342cd97834d37a2e908d6212a9d14; no voters:
I20260928 22:48:19.845956 19242 raft_consensus.cc:3037] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:19.846004 19440 raft_consensus.cc:2781] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:19.846127 19440 raft_consensus.cc:686] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 1 LEADER]: Becoming Leader. State: Replica: da5342cd97834d37a2e908d6212a9d14, State: Running, Role: LEADER
I20260928 22:48:19.846222 19440 consensus_queue.cc:237] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } }
I20260928 22:48:19.846752 19242 raft_consensus.cc:2454] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5342cd97834d37a2e908d6212a9d14 in term 1.
I20260928 22:48:19.847033 18969 catalog_manager.cc:5495] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 reported cstate change: term changed from 0 to 1, leader changed from <none> to da5342cd97834d37a2e908d6212a9d14 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "da5342cd97834d37a2e908d6212a9d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5342cd97834d37a2e908d6212a9d14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20260928 22:48:19.877204 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.875081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.0.98.1:34711)
0928 22:48:19.875139 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:19.875141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:19.875149 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:19.875184 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:19.875185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:19.875209 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:19.875268 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:19.875380 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.875845 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.876562 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:19.876768 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:19.876818 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:19.876924 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:19.877016 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:19.877082 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:19.877117 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 22:48:19.877126 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:48:19.879072 19242 raft_consensus.cc:1261] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 1 FOLLOWER]: Refusing update from remote peer da5342cd97834d37a2e908d6212a9d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:19.879388 19373 raft_consensus.cc:1261] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 1 FOLLOWER]: Refusing update from remote peer da5342cd97834d37a2e908d6212a9d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:19.879400 19443 consensus_queue.cc:1035] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d5f1287cff41799f23826f352dfa13" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:19.879647 19443 consensus_queue.cc:1035] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a99f5734e44abfb1c6bc3cd6706219" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:19.883193 19447 mvcc.cc:205] Tried to move back new op lower bound from 7334443826702225408 to 7334443826570878976. Current Snapshot: MvccSnapshot[applied={T|T < 7334443826702225408}]
I20260928 22:48:19.884521 18969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:19.884608 18969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:19.885344 18969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f2ec221ca74dd7a0b394876c359001: Sending DeleteTablet for 3 replicas of tablet 73d90342cb284a6294d8592e1eddcd1f
I20260928 22:48:19.885619 19091 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d90342cb284a6294d8592e1eddcd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='slave'} at 127.0.0.1:52232
I20260928 22:48:19.885681 19222 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d90342cb284a6294d8592e1eddcd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='slave'} at 127.0.0.1:39082
I20260928 22:48:19.885767 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d90342cb284a6294d8592e1eddcd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:19 UTC) from {username='slave'} at 127.0.0.1:33712
I20260928 22:48:19.885932 19457 tablet_replica.cc:330] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: stopping tablet replica
I20260928 22:48:19.886001 19456 tablet_replica.cc:330] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14: stopping tablet replica
I20260928 22:48:19.886090 19456 raft_consensus.cc:2229] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:19.886098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19029
I20260928 22:48:19.886101 19457 raft_consensus.cc:2229] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:19.886211 19456 raft_consensus.cc:2258] T 73d90342cb284a6294d8592e1eddcd1f P da5342cd97834d37a2e908d6212a9d14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:19.886330 19457 raft_consensus.cc:2258] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:19.887777 19458 tablet_replica.cc:330] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: stopping tablet replica
I20260928 22:48:19.887880 19458 raft_consensus.cc:2229] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:19.887982 19458 raft_consensus.cc:2258] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:19.888000 19457 ts_tablet_manager.cc:1905] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:19.888370 19458 ts_tablet_manager.cc:1905] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:19.889802 19458 ts_tablet_manager.cc:1918] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:19.889803 19457 ts_tablet_manager.cc:1918] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:19.889868 19457 log.cc:1196] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-1/wal/wals/73d90342cb284a6294d8592e1eddcd1f
I20260928 22:48:19.889868 19458 log.cc:1196] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/73d90342cb284a6294d8592e1eddcd1f
I20260928 22:48:19.890125 19458 ts_tablet_manager.cc:1939] T 73d90342cb284a6294d8592e1eddcd1f P 23a99f5734e44abfb1c6bc3cd6706219: Deleting consensus metadata
I20260928 22:48:19.890125 19457 ts_tablet_manager.cc:1939] T 73d90342cb284a6294d8592e1eddcd1f P 27d5f1287cff41799f23826f352dfa13: Deleting consensus metadata
I20260928 22:48:19.890506 18956 catalog_manager.cc:4841] TS 23a99f5734e44abfb1c6bc3cd6706219 (127.0.98.3:37461): tablet 73d90342cb284a6294d8592e1eddcd1f (table test-table [id=ccc52ec3b2e74cc89bd13e0985a017ff]) successfully deleted
I20260928 22:48:19.890553 18955 catalog_manager.cc:4841] TS 27d5f1287cff41799f23826f352dfa13 (127.0.98.2:45031): tablet 73d90342cb284a6294d8592e1eddcd1f (table test-table [id=ccc52ec3b2e74cc89bd13e0985a017ff]) successfully deleted
W20260928 22:48:19.891552 18953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34711 (error 108)
I20260928 22:48:19.891758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19160
I20260928 22:48:19.892014 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:19.891896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34711 (local address 127.0.0.1:52242)
0928 22:48:19.891942 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:19.891973 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:48:19.892107 18953 catalog_manager.cc:4584] TS da5342cd97834d37a2e908d6212a9d14 (127.0.98.1:34711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73d90342cb284a6294d8592e1eddcd1f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34711: connect: Connection refused (error 111)
I20260928 22:48:19.896409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19291
I20260928 22:48:19.900493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 18938
2026-09-28T22:48:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:19 dist-test-slave-pxr1 krb5kdc[19462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:19 dist-test-slave-pxr1 krb5kdc[19462](info): set up 2 sockets
Sep 28 22:48:19 dist-test-slave-pxr1 krb5kdc[19462](info): commencing operation
krb5kdc: starting...
W20260928 22:48:20.860261 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:20 dist-test-slave-pxr1 krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:20.970809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:21.050135 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:21.050357 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:21.050390 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:21.051590 19475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:21.051637 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:21.051656 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:21.051671 19475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:21.051687 19475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:21.053194 19475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:21.053449 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:21.053668 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:21.058264 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:21.058357 19475 server_base.cc:1034] running on GCE node
W20260928 22:48:21.058235 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.058266 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:21.058624 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:21.058895 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:21.060035 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790635701060021 us; error 28 us; skew 500 ppm
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:21.062438 19475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:21.062845 19475 webserver.cc:460] Webserver started at http://127.0.98.62:34145/ using document root <none> and password file <none>
I20260928 22:48:21.063054 19475 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:21.063097 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:21.063218 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:21.064244 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "5c9521bb66854ef1a26a2cd83c2508af"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c5c0c242b3b9468ad61f14ce0d33df28"
server_key_iv: "5a1f1bf8a28e824e6c735a1f3ec9d9ff"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.064563 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c9521bb66854ef1a26a2cd83c2508af"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c5c0c242b3b9468ad61f14ce0d33df28"
server_key_iv: "5a1f1bf8a28e824e6c735a1f3ec9d9ff"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.065871 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:21.066495 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.066623 19475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:21.066704 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "5c9521bb66854ef1a26a2cd83c2508af"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c5c0c242b3b9468ad61f14ce0d33df28"
server_key_iv: "5a1f1bf8a28e824e6c735a1f3ec9d9ff"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.066762 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:21.095996 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:21.098161 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:21.098300 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:21.102155 19475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36861
I20260928 22:48:21.102191 19542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36861 every 8 connection(s)
I20260928 22:48:21.102530 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:21.103503 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:21.105151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19475
I20260928 22:48:21.105273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:21.105468 392 external_mini_cluster.cc:1428] Setting key efeae86899936ca0fc353ee42719f502
I20260928 22:48:21.106180 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: Bootstrap starting.
I20260928 22:48:21.106750 19543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:21.107071 19543 log.cc:824] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:21.107784 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: No bootstrap required, opened a new log
I20260928 22:48:21.109881 19543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260928 22:48:21.110031 19543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:21.110064 19543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9521bb66854ef1a26a2cd83c2508af, State: Initialized, Role: FOLLOWER
I20260928 22:48:21.110219 19543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260928 22:48:21.110306 19543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:21.110361 19543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:21.110420 19543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:21.111148 19543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260928 22:48:21.111269 19543 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9521bb66854ef1a26a2cd83c2508af; no voters:
I20260928 22:48:21.111450 19543 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:21.111495 19548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:21.111657 19548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [term 1 LEADER]: Becoming Leader. State: Replica: 5c9521bb66854ef1a26a2cd83c2508af, State: Running, Role: LEADER
I20260928 22:48:21.111716 19543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:21.111778 19548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260928 22:48:21.112092 19548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9521bb66854ef1a26a2cd83c2508af. Latest consensus state: current_term: 1 leader_uuid: "5c9521bb66854ef1a26a2cd83c2508af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } } }
I20260928 22:48:21.112174 19548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [sys.catalog]: This master's current role is: LEADER
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:21.112102 19550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9521bb66854ef1a26a2cd83c2508af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9521bb66854ef1a26a2cd83c2508af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } } }
I20260928 22:48:21.112340 19550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af [sys.catalog]: This master's current role is: LEADER
W20260928 22:48:21.113903 19563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:21.113963 19563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:21.114008 19555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:21.114163 19555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:21.115769 19555 catalog_manager.cc:1347] Generated new cluster ID: cb9f192796d646888af78d8ea6d77a58
I20260928 22:48:21.115821 19555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:21.116572 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.105995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.0.98.62:36861)
0928 22:48:21.106166 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.106174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.106199 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:48:21.106653 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.106660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.106708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.106844 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:21.107593 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.108515 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.109590 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.109987 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.110081 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.112848 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:21.112852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:21.112852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:21.112866 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:21.114791 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.115119 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.115122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.115122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.115241 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.115491 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.115492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.115492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.115805 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:21.115948 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.116196 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.116348 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260928 22:48:21.133255 19555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:21.134066 19555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:21.156634 19555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: Generated new TSK 0
I20260928 22:48:21.156895 19555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 22:48:21.181704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:21.262730 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:21.262935 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:21.262959 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:21.264143 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:21.264189 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:21.264215 19569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:21.265767 19569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:21.265959 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:21.266180 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:21.271006 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.271051 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.270985 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:21.271127 19569 server_base.cc:1034] running on GCE node
I20260928 22:48:21.271399 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:21.271601 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:21.272749 19569 hybrid_clock.cc:648] HybridClock initialized: now 1790635701272734 us; error 31 us; skew 500 ppm
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:21.275142 19569 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 22:48:21.275559 19569 webserver.cc:460] Webserver started at http://127.0.98.1:36225/ using document root <none> and password file <none>
I20260928 22:48:21.275770 19569 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:21.275815 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:21.275940 19569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:21.277034 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "36e9049a8795419b8db4ca94519d12b7"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "99b6885667d4130b055de82f18a51e8b"
server_key_iv: "06b36cea8e7ac98f6e53edaa27694189"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.277355 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36e9049a8795419b8db4ca94519d12b7"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "99b6885667d4130b055de82f18a51e8b"
server_key_iv: "06b36cea8e7ac98f6e53edaa27694189"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.278533 19569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:21.279170 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.279320 19569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.279392 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "36e9049a8795419b8db4ca94519d12b7"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "99b6885667d4130b055de82f18a51e8b"
server_key_iv: "06b36cea8e7ac98f6e53edaa27694189"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.279457 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:21.303179 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:21.304920 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:21.305063 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:21.305281 19569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:21.305649 19569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:21.305684 19569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.305714 19569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:21.305732 19569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.311000 19569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38745
I20260928 22:48:21.311074 19697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38745 every 8 connection(s)
I20260928 22:48:21.311416 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:21.316406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19569
I20260928 22:48:21.316527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:21.316776 392 external_mini_cluster.cc:1428] Setting key b39ca27c4dfe39212f77c205328f34a1
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:21.320688 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.312081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58509 (local address 127.0.98.62:36861)
0928 22:48:21.312134 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.312136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.312367 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:48:21.313352 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.313353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.313366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.313416 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:21.314036 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.314540 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.315542 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.315759 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.315816 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.318152 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:21.318153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:21.318153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:21.318161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:21.319452 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.319809 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.319810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.319811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.319862 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.320069 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.320070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.320070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.320182 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:21.320237 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.320501 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.320589 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:48:21.320878 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.312063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.1:58509)
0928 22:48:21.312219 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.312265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.313246 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.313483 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.313487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.313749 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:21.313936 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.313941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.314623 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.314624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.315458 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.315460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.315829 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.315830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.315953 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.316445 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:21.316455 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:21.318064 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:21.319528 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.319536 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.319539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.319741 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.319916 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.319916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.319917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.320012 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.320237 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:21.320243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:21.320379 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.320472 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.320636 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260928 22:48:21.321466 19698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260928 22:48:21.321554 19698 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:21.321722 19698 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260928 22:48:21.322274 19507 ts_manager.cc:194] Registered new tserver with Master: 36e9049a8795419b8db4ca94519d12b7 (127.0.98.1:38745)
I20260928 22:48:21.323194 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58509
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 22:48:21.341661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:21.420244 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:21.420470 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:21.420495 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:21.421722 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:21.421775 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:21.421802 19703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:21.423297 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:21.423503 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:21.423689 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:21.428254 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.428342 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.428359 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:21.428596 19703 server_base.cc:1034] running on GCE node
I20260928 22:48:21.428826 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:21.429046 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:21.430176 19703 hybrid_clock.cc:648] HybridClock initialized: now 1790635701430156 us; error 35 us; skew 500 ppm
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:21.432613 19703 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 22:48:21.433061 19703 webserver.cc:460] Webserver started at http://127.0.98.2:41027/ using document root <none> and password file <none>
I20260928 22:48:21.433259 19703 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:21.433305 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:21.433424 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:21.434432 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "60c7171ceadc4903aff25d19239313f6"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "495ece089ba02ebc497811e7d069f593"
server_key_iv: "34168e6c354c3ec1329762b546a1466d"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.434764 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60c7171ceadc4903aff25d19239313f6"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "495ece089ba02ebc497811e7d069f593"
server_key_iv: "34168e6c354c3ec1329762b546a1466d"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.436002 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:21.436640 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.436810 19703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.436892 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "60c7171ceadc4903aff25d19239313f6"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "495ece089ba02ebc497811e7d069f593"
server_key_iv: "34168e6c354c3ec1329762b546a1466d"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.436954 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:21.445340 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:21.447331 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:21.447461 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:21.447660 19703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:21.447937 19703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:21.447973 19703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.448001 19703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:21.448019 19703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.453459 19703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37295
I20260928 22:48:21.453521 19831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37295 every 8 connection(s)
I20260928 22:48:21.453822 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:21.455977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19703
I20260928 22:48:21.456085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:21.456250 392 external_mini_cluster.cc:1428] Setting key 6374e422b18a049663523bcdfa43dfb9
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:21.463308 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.454515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60573 (local address 127.0.98.62:36861)
0928 22:48:21.454562 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.454564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.454748 (+ 184us) server_negotiation.cc:407] Connection header received
0928 22:48:21.455730 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.455731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.455743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.455795 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:21.456469 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.457170 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.458151 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.458335 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.458390 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.460553 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:21.460555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:21.460555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:21.460563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:21.462000 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.462405 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.462406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.462407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.462462 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.462688 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.462689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.462689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.462809 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:21.462865 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.463133 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.463217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:48:21.463506 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.454472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.2:60573)
0928 22:48:21.454634 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.454670 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.455628 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.455864 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.455868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.456151 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:21.456371 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.456376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.457257 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.457258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.458069 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.458071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.458401 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.458402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.458527 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.459000 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:21.459011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:21.460455 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:21.462095 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.462104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.462107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.462335 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.462522 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.462523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.462523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.462632 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.462864 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:21.462871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:21.463015 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.463106 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.463269 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 22:48:21.463971 19832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260928 22:48:21.464074 19832 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:21.464247 19832 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260928 22:48:21.464617 19507 ts_manager.cc:194] Registered new tserver with Master: 60c7171ceadc4903aff25d19239313f6 (127.0.98.2:37295)
I20260928 22:48:21.465011 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 22:48:21.479542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:21.558259 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:21.558467 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:21.558490 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:21.559640 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:21.559682 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:21.559707 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:21.561177 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:21.561379 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:21.561563 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:21.566285 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.566298 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:21.566294 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:21.566313 19837 server_base.cc:1034] running on GCE node
I20260928 22:48:21.566728 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:21.566962 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:21.568099 19837 hybrid_clock.cc:648] HybridClock initialized: now 1790635701568086 us; error 30 us; skew 500 ppm
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:21.570534 19837 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 22:48:21.570979 19837 webserver.cc:460] Webserver started at http://127.0.98.3:41977/ using document root <none> and password file <none>
I20260928 22:48:21.571189 19837 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:21.571235 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:21.571364 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:21.572391 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "7183d2ced6c142b6864f7c96d75e2f47"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c369102407f685419367e2bc3115b69f"
server_key_iv: "7505d5c1435b0d21dede7b19231833f7"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.572781 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7183d2ced6c142b6864f7c96d75e2f47"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c369102407f685419367e2bc3115b69f"
server_key_iv: "7505d5c1435b0d21dede7b19231833f7"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.574007 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:21.574671 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.574823 19837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:21.574903 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "7183d2ced6c142b6864f7c96d75e2f47"
format_stamp: "Formatted at 2026-09-28 22:48:21 on dist-test-slave-pxr1"
server_key: "c369102407f685419367e2bc3115b69f"
server_key_iv: "7505d5c1435b0d21dede7b19231833f7"
server_key_version: "encryptionkey@0"
I20260928 22:48:21.574970 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:21.597321 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:21.599510 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:21.599650 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:21.599858 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:21.600148 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:21.600183 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.600211 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:21.600235 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:21.605801 19837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42095
I20260928 22:48:21.605850 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42095 every 8 connection(s)
I20260928 22:48:21.606154 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:48:21 dist-test-slave-pxr1 krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:21.614138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 19837
I20260928 22:48:21.614260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:21.614459 392 external_mini_cluster.cc:1428] Setting key e9433a0e2ddcaf6bb94dc8961b3f9cb5
I20260928 22:48:21.615518 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.606796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44473 (local address 127.0.98.62:36861)
0928 22:48:21.606858 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.606860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.607062 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:48:21.607986 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.607988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.607998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.608055 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:21.608662 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.609203 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.610181 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.610356 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.610401 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.612634 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:21.612635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:21.612636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:21.612643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:21.613981 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.614469 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.614469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.614470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.614529 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.614804 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.614805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.614805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.614959 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:21.615025 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.615334 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.615453 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:21.615670 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.606770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.3:44473)
0928 22:48:21.606937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.606978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.607903 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.608116 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.608120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.608379 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:21.608595 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.608599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.609280 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.609281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.610116 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.610119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.610447 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.610448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.610557 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.611009 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:21.611017 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:21.612543 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:21.614050 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.614059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.614061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.614360 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.614586 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:21.614587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:21.614588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:21.614729 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:21.615024 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:21.615030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:21.615198 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.615304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.615470 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 22:48:21.616068 19966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260928 22:48:21.616166 19966 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:21.616355 19966 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260928 22:48:21.616698 19507 ts_manager.cc:194] Registered new tserver with Master: 7183d2ced6c142b6864f7c96d75e2f47 (127.0.98.3:42095)
I20260928 22:48:21.617091 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44473
I20260928 22:48:21.626250 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:21.632846 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.627562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.0.98.62:36861)
0928 22:48:21.627629 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.627631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.627684 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:48:21.627741 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.627741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.627750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.627796 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:21.628280 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.628806 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.629469 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.629646 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.629691 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.630815 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:21.630816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:21.630816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:21.630820 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:21.631912 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.632210 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.632210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.632210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.632254 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:21.632379 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:21.632380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:21.632380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:21.632517 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:21.632586 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.632680 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.632773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:48:21.634321 19507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:21.635821 19507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:21.641734 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.637834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.0.98.3:42095)
0928 22:48:21.638148 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.638153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.638162 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:21.638198 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.638199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.638210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.638274 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:21.638422 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.639362 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.640843 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.641335 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.641403 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.641473 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.641603 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.641646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":73,"threads_started":1}
I20260928 22:48:21.641731 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.637737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42095 (local address 127.0.0.1:49912)
0928 22:48:21.637897 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.637940 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.638003 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.638289 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.638292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.638294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:21.638361 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.638363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.639460 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.639461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.640773 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.640776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.641414 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.641415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.641499 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.641514 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.641582 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.641664 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":24,"thread_start_us":57,"threads_started":1}
I20260928 22:48:21.642118 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.637941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37295 (local address 127.0.0.1:37986)
0928 22:48:21.638038 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.638047 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.638099 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.638484 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.638484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.638487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:21.638532 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.638533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.639816 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.639818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.640897 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.640899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.641776 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.641777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.641959 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.641964 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.641991 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.642030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":36,"threads_started":1}
I20260928 22:48:21.642436 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.638122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.0.98.2:37295)
0928 22:48:21.638285 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.638288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.638298 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:48:21.638340 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.638342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.638356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.638601 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:21.638672 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.639709 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.640958 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.641680 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.642220 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.642291 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.642316 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.642359 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 22:48:21.642474 19900 tablet_service.cc:1467] Processing CreateTablet for tablet faaacb20ac5b411290cd007dc3469f91 (DEFAULT_TABLE table=test-table [id=73521d21f3264c06a84910ba37847525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:21.642838 19900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaacb20ac5b411290cd007dc3469f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:21.643059 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.637754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38745 (local address 127.0.0.1:39180)
0928 22:48:21.638007 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.638016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.638054 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.638451 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.638451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.638453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:21.638492 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.638493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.640971 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.640972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.642031 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.642033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.642846 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.642847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.642912 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.642915 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.642964 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.643002 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":61,"threads_started":1}
I20260928 22:48:21.643106 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.637877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.98.1:38745)
0928 22:48:21.638322 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.638326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.638334 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:21.638369 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.638371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.638385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.638456 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:21.638544 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.639707 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.642116 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.642759 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.642823 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.642875 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.642989 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.643033 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":74,"threads_started":1}
I20260928 22:48:21.645426 19980 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Bootstrap starting.
I20260928 22:48:21.645363 19632 tablet_service.cc:1467] Processing CreateTablet for tablet faaacb20ac5b411290cd007dc3469f91 (DEFAULT_TABLE table=test-table [id=73521d21f3264c06a84910ba37847525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:21.645498 19766 tablet_service.cc:1467] Processing CreateTablet for tablet faaacb20ac5b411290cd007dc3469f91 (DEFAULT_TABLE table=test-table [id=73521d21f3264c06a84910ba37847525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:21.645691 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaacb20ac5b411290cd007dc3469f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:21.645774 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaacb20ac5b411290cd007dc3469f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:21.645982 19980 tablet_bootstrap.cc:654] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:21.646525 19980 log.cc:824] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:21.647435 19980 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: No bootstrap required, opened a new log
I20260928 22:48:21.647482 19980 ts_tablet_manager.cc:1397] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:21.648862 19983 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Bootstrap starting.
I20260928 22:48:21.648922 19980 raft_consensus.cc:348] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.649097 19980 raft_consensus.cc:374] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:21.649142 19980 raft_consensus.cc:729] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7183d2ced6c142b6864f7c96d75e2f47, State: Initialized, Role: FOLLOWER
I20260928 22:48:21.649269 19980 consensus_queue.cc:260] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.649535 19980 ts_tablet_manager.cc:1428] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:21.649549 19983 tablet_bootstrap.cc:654] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:21.649561 19966 heartbeater.cc:503] Master 127.0.98.62:36861 was elected leader, sending a full tablet report...
I20260928 22:48:21.650027 19983 log.cc:824] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:21.650336 19982 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Bootstrap starting.
I20260928 22:48:21.650701 19983 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: No bootstrap required, opened a new log
I20260928 22:48:21.650766 19983 ts_tablet_manager.cc:1397] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:48:21.650959 19982 tablet_bootstrap.cc:654] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:21.651510 19982 log.cc:824] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:21.652254 19982 tablet_bootstrap.cc:492] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: No bootstrap required, opened a new log
I20260928 22:48:21.652323 19982 ts_tablet_manager.cc:1397] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:21.652695 19983 raft_consensus.cc:348] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.652940 19983 raft_consensus.cc:374] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:21.652979 19983 raft_consensus.cc:729] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c7171ceadc4903aff25d19239313f6, State: Initialized, Role: FOLLOWER
I20260928 22:48:21.653096 19983 consensus_queue.cc:260] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.653375 19832 heartbeater.cc:503] Master 127.0.98.62:36861 was elected leader, sending a full tablet report...
I20260928 22:48:21.653391 19983 ts_tablet_manager.cc:1428] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:48:21.654052 19982 raft_consensus.cc:348] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.654179 19982 raft_consensus.cc:374] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:21.654201 19982 raft_consensus.cc:729] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e9049a8795419b8db4ca94519d12b7, State: Initialized, Role: FOLLOWER
I20260928 22:48:21.654305 19982 consensus_queue.cc:260] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.654538 19982 ts_tablet_manager.cc:1428] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:48:21.654570 19698 heartbeater.cc:503] Master 127.0.98.62:36861 was elected leader, sending a full tablet report...
W20260928 22:48:21.704344 19833 tablet.cc:2369] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:21.812067 19699 tablet.cc:2369] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:21.856673 19967 tablet.cc:2369] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:21.903914 19987 raft_consensus.cc:482] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:21.904073 19987 raft_consensus.cc:504] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.904457 19987 leader_election.cc:288] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7183d2ced6c142b6864f7c96d75e2f47 (127.0.98.3:42095), 36e9049a8795419b8db4ca94519d12b7 (127.0.98.1:38745)
I20260928 22:48:21.907194 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.904710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45341 (local address 127.0.98.1:38745)
0928 22:48:21.904772 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.904774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.904964 (+ 190us) server_negotiation.cc:407] Connection header received
0928 22:48:21.905020 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.905021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.905032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.905079 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:21.905211 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.905670 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906547 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.906899 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906956 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.906965 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.907093 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.907129 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:48:21.907202 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.904811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33353 (local address 127.0.98.3:42095)
0928 22:48:21.904882 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.904884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.904893 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:21.904918 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.904919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.904927 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.904972 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:21.905144 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.905623 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906529 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.906896 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906954 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.906964 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.907098 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.907132 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:48:21.907212 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.904646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42095 (local address 127.0.98.2:33353)
0928 22:48:21.904689 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.904701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.904763 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.904994 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.904995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.904997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:21.905061 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.905063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.905697 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.905698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.906435 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.906437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.906970 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.907040 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.907042 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.907094 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.907132 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:48:21.907220 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.904646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38745 (local address 127.0.98.2:45341)
0928 22:48:21.904887 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.904896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:21.904964 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:21.905086 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:21.905087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:21.905088 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:21.905154 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.905156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.905737 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.905738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.906463 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:21.906464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.906965 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.906965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:21.907034 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.907037 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:21.907081 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:48:21.907118 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":50,"thread_start_us":123,"threads_started":1}
I20260928 22:48:21.907559 19920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaacb20ac5b411290cd007dc3469f91" candidate_uuid: "60c7171ceadc4903aff25d19239313f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7183d2ced6c142b6864f7c96d75e2f47" is_pre_election: true
I20260928 22:48:21.907706 19920 raft_consensus.cc:2454] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c7171ceadc4903aff25d19239313f6 in term 0.
I20260928 22:48:21.907712 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaacb20ac5b411290cd007dc3469f91" candidate_uuid: "60c7171ceadc4903aff25d19239313f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e9049a8795419b8db4ca94519d12b7" is_pre_election: true
I20260928 22:48:21.907867 19652 raft_consensus.cc:2454] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c7171ceadc4903aff25d19239313f6 in term 0.
I20260928 22:48:21.907938 19719 leader_election.cc:302] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c7171ceadc4903aff25d19239313f6, 7183d2ced6c142b6864f7c96d75e2f47; no voters:
I20260928 22:48:21.908077 19987 raft_consensus.cc:2781] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:21.908144 19987 raft_consensus.cc:482] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:21.908167 19987 raft_consensus.cc:3037] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:21.908845 19987 raft_consensus.cc:504] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.908964 19987 leader_election.cc:288] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [CANDIDATE]: Term 1 election: Requested vote from peers 7183d2ced6c142b6864f7c96d75e2f47 (127.0.98.3:42095), 36e9049a8795419b8db4ca94519d12b7 (127.0.98.1:38745)
I20260928 22:48:21.909118 19920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaacb20ac5b411290cd007dc3469f91" candidate_uuid: "60c7171ceadc4903aff25d19239313f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7183d2ced6c142b6864f7c96d75e2f47"
I20260928 22:48:21.909119 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaacb20ac5b411290cd007dc3469f91" candidate_uuid: "60c7171ceadc4903aff25d19239313f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e9049a8795419b8db4ca94519d12b7"
I20260928 22:48:21.909190 19920 raft_consensus.cc:3037] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:21.909193 19652 raft_consensus.cc:3037] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:21.909845 19920 raft_consensus.cc:2454] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c7171ceadc4903aff25d19239313f6 in term 1.
I20260928 22:48:21.909855 19652 raft_consensus.cc:2454] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c7171ceadc4903aff25d19239313f6 in term 1.
I20260928 22:48:21.909993 19720 leader_election.cc:302] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e9049a8795419b8db4ca94519d12b7, 60c7171ceadc4903aff25d19239313f6; no voters:
I20260928 22:48:21.910097 19987 raft_consensus.cc:2781] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:21.910234 19987 raft_consensus.cc:686] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 1 LEADER]: Becoming Leader. State: Replica: 60c7171ceadc4903aff25d19239313f6, State: Running, Role: LEADER
I20260928 22:48:21.910306 19987 consensus_queue.cc:237] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } }
I20260928 22:48:21.911027 19507 catalog_manager.cc:5495] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60c7171ceadc4903aff25d19239313f6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "60c7171ceadc4903aff25d19239313f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c7171ceadc4903aff25d19239313f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } health_report { overall_health: HEALTHY } } }
I20260928 22:48:21.926048 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.923729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.0.98.2:37295)
0928 22:48:21.923796 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:21.923797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:21.923806 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:21.923840 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:21.923840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:21.923860 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:21.923919 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:21.924021 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.924493 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.925351 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:21.925547 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:21.925602 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:21.925743 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:21.925843 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:21.925915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:21.925932 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 22:48:21.925952 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:48:21.928062 19652 raft_consensus.cc:1261] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 1 FOLLOWER]: Refusing update from remote peer 60c7171ceadc4903aff25d19239313f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:21.928087 19920 raft_consensus.cc:1261] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 1 FOLLOWER]: Refusing update from remote peer 60c7171ceadc4903aff25d19239313f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:21.928419 19987 consensus_queue.cc:1035] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e9049a8795419b8db4ca94519d12b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:21.928660 19987 consensus_queue.cc:1035] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7183d2ced6c142b6864f7c96d75e2f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:21.931454 19995 mvcc.cc:205] Tried to move back new op lower bound from 7334443835094626304 to 7334443835025219584. Current Snapshot: MvccSnapshot[applied={T|T < 7334443835094626304}]
I20260928 22:48:21.931728 19994 mvcc.cc:205] Tried to move back new op lower bound from 7334443835094626304 to 7334443835025219584. Current Snapshot: MvccSnapshot[applied={T|T < 7334443835094626304}]
I20260928 22:48:21.934275 19506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:21.934437 19506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:21.935312 19506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9521bb66854ef1a26a2cd83c2508af: Sending DeleteTablet for 3 replicas of tablet faaacb20ac5b411290cd007dc3469f91
I20260928 22:48:21.935631 19900 tablet_service.cc:1514] Processing DeleteTablet for tablet faaacb20ac5b411290cd007dc3469f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49912
I20260928 22:48:21.935652 19766 tablet_service.cc:1514] Processing DeleteTablet for tablet faaacb20ac5b411290cd007dc3469f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37986
I20260928 22:48:21.935685 19632 tablet_service.cc:1514] Processing DeleteTablet for tablet faaacb20ac5b411290cd007dc3469f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39180
I20260928 22:48:21.935904 20005 tablet_replica.cc:330] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7: stopping tablet replica
I20260928 22:48:21.936007 20005 raft_consensus.cc:2229] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:21.936120 20005 raft_consensus.cc:2258] T faaacb20ac5b411290cd007dc3469f91 P 36e9049a8795419b8db4ca94519d12b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:21.936235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19569
I20260928 22:48:21.941206 20003 tablet_replica.cc:330] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: stopping tablet replica
I20260928 22:48:21.941313 20003 raft_consensus.cc:2229] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:21.941416 20003 raft_consensus.cc:2258] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:21.941584 20004 tablet_replica.cc:330] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: stopping tablet replica
I20260928 22:48:21.941694 20004 raft_consensus.cc:2229] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:21.941797 20004 raft_consensus.cc:2258] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:21.942239 20003 ts_tablet_manager.cc:1905] T faaacb20ac5b411290cd007dc3469f91 P 60c7171ceadc4903aff25d19239313f6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:48:21.943353 19492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38745 (error 108)
I20260928 22:48:21.943861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19703
I20260928 22:48:21.945072 20004 ts_tablet_manager.cc:1905] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:21.945616 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.945475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38745 (local address 127.0.0.1:39182)
0928 22:48:21.945535 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.945578 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 22:48:21.945734 19492 catalog_manager.cc:4584] TS 36e9049a8795419b8db4ca94519d12b7 (127.0.98.1:38745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet faaacb20ac5b411290cd007dc3469f91: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38745: connect: Connection refused (error 111)
I20260928 22:48:21.946511 20004 ts_tablet_manager.cc:1918] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:21.946579 20004 log.cc:1196] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/faaacb20ac5b411290cd007dc3469f91
I20260928 22:48:21.946795 20004 ts_tablet_manager.cc:1939] T faaacb20ac5b411290cd007dc3469f91 P 7183d2ced6c142b6864f7c96d75e2f47: Deleting consensus metadata
I20260928 22:48:21.947172 19491 catalog_manager.cc:4841] TS 7183d2ced6c142b6864f7c96d75e2f47 (127.0.98.3:42095): tablet faaacb20ac5b411290cd007dc3469f91 (table test-table [id=73521d21f3264c06a84910ba37847525]) successfully deleted
I20260928 22:48:21.948774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19837
I20260928 22:48:21.948825 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:21.948702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37295 (local address 127.0.0.1:38008)
0928 22:48:21.948769 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:21.948796 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:48:21.952955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 19475
2026-09-28T22:48:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T22:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:21Z Disabled control of system clock
I20260928 22:48:21.978849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42413
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:22.057631 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:22.057869 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:22.057897 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:22.059098 20010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:22.059145 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:22.059161 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:22.059178 20010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:22.059193 20010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:22.060900 20010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:22.061180 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:22.061476 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:22.066141 20015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.066159 20010 server_base.cc:1034] running on GCE node
W20260928 22:48:22.066252 20018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.066166 20016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.066624 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:22.066905 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:22.068074 20010 hybrid_clock.cc:648] HybridClock initialized: now 1790635702068056 us; error 30 us; skew 500 ppm
I20260928 22:48:22.069435 20010 webserver.cc:460] Webserver started at http://127.0.98.62:43827/ using document root <none> and password file <none>
I20260928 22:48:22.069674 20010 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:22.069731 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:22.069864 20010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:22.070938 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "64035475ff62477eb3f08ea871c1fb1e"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "875c3580364eb3d8e2fc073d8cdf480b"
server_key_iv: "ed2808a497637b0503a08bb2bd708e25"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.071305 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "64035475ff62477eb3f08ea871c1fb1e"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "875c3580364eb3d8e2fc073d8cdf480b"
server_key_iv: "ed2808a497637b0503a08bb2bd708e25"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.072602 20010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:22.073378 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.073535 20010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:22.073623 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "64035475ff62477eb3f08ea871c1fb1e"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "875c3580364eb3d8e2fc073d8cdf480b"
server_key_iv: "ed2808a497637b0503a08bb2bd708e25"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.073696 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:22.086548 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:22.086876 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:22.087028 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:22.090981 20010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42413
I20260928 22:48:22.091017 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42413 every 8 connection(s)
I20260928 22:48:22.091367 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:22.092399 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:22.092916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20010
I20260928 22:48:22.093020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:22.093211 392 external_mini_cluster.cc:1428] Setting key ad761faa1c6499f2c8d62d17a6f56221
I20260928 22:48:22.095364 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: Bootstrap starting.
I20260928 22:48:22.095988 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:22.096241 20077 log.cc:824] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:22.096948 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: No bootstrap required, opened a new log
I20260928 22:48:22.098929 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } }
I20260928 22:48:22.099097 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:22.099135 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64035475ff62477eb3f08ea871c1fb1e, State: Initialized, Role: FOLLOWER
I20260928 22:48:22.099303 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.093761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.0.98.62:42413)
0928 22:48:22.094077 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.094085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.094118 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:48:22.094728 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.094736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.094803 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.094937 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:22.095177 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.096355 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.097197 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.097613 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.097726 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.098602 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:22.098606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:22.098606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:22.098777 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:22.098811 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:22.098822 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:22.098877 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.098966 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.099063 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":147,"threads_started":1}
I20260928 22:48:22.099270 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } }
I20260928 22:48:22.099365 20077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:22.099397 20077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:22.099436 20077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:22.100167 20077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } }
I20260928 22:48:22.100267 20077 leader_election.cc:302] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64035475ff62477eb3f08ea871c1fb1e; no voters:
I20260928 22:48:22.100402 20077 leader_election.cc:288] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:22.100489 20082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:22.100636 20082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [term 1 LEADER]: Becoming Leader. State: Replica: 64035475ff62477eb3f08ea871c1fb1e, State: Running, Role: LEADER
I20260928 22:48:22.100651 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:22.100754 20082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } }
I20260928 22:48:22.101119 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64035475ff62477eb3f08ea871c1fb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } } }
I20260928 22:48:22.101130 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64035475ff62477eb3f08ea871c1fb1e. Latest consensus state: current_term: 1 leader_uuid: "64035475ff62477eb3f08ea871c1fb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64035475ff62477eb3f08ea871c1fb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42413 } } }
I20260928 22:48:22.101292 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:22.101495 20091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:22.101684 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:22.101969 20091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:22.103618 20091 catalog_manager.cc:1347] Generated new cluster ID: bc8fa461363240378f1078eed4681108
I20260928 22:48:22.103677 20091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:22.107695 20091 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:22.108479 20091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:22.112870 20091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: Generated new TSK 0
I20260928 22:48:22.113065 20091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:22.119568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:48:22.199554 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:22.199772 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:22.199801 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:22.201102 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:22.201144 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:22.201179 20101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 22:48:22.202631 20101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:22.202880 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:22.203166 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:22.207805 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.207908 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.207805 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.208122 20101 server_base.cc:1034] running on GCE node
I20260928 22:48:22.208310 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:22.208544 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:22.209697 20101 hybrid_clock.cc:648] HybridClock initialized: now 1790635702209664 us; error 47 us; skew 500 ppm
I20260928 22:48:22.210932 20101 webserver.cc:460] Webserver started at http://127.0.98.1:38663/ using document root <none> and password file <none>
I20260928 22:48:22.211179 20101 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:22.211228 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:22.211400 20101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:22.212479 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data/instance:
uuid: "3554b6cafd4846ca9d55987d654617c7"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "046a64380e7a14d7086b8f33bddafe42"
server_key_iv: "f8b7a8a445f3c204926e3cb9f6e65d69"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.212849 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3554b6cafd4846ca9d55987d654617c7"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "046a64380e7a14d7086b8f33bddafe42"
server_key_iv: "f8b7a8a445f3c204926e3cb9f6e65d69"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.214188 20101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:22.215029 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.215195 20101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:22.215281 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal
uuid: "3554b6cafd4846ca9d55987d654617c7"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "046a64380e7a14d7086b8f33bddafe42"
server_key_iv: "f8b7a8a445f3c204926e3cb9f6e65d69"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.215354 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:22.224994 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:22.225281 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:22.225405 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:22.225622 20101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:22.225929 20101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:22.225966 20101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.225997 20101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:22.226022 20101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.231547 20101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33275
I20260928 22:48:22.231599 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33275 every 8 connection(s)
I20260928 22:48:22.231910 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:22.234768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20101
I20260928 22:48:22.234877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:22.235064 392 external_mini_cluster.cc:1428] Setting key 2e404e1224503efd2241a51997f0d468
I20260928 22:48:22.236052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:48:22.237495 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.232610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33641 (local address 127.0.98.62:42413)
0928 22:48:22.232672 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.232674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.232925 (+ 251us) server_negotiation.cc:407] Connection header received
0928 22:48:22.233480 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.233481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.233495 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.233540 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:22.234251 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.234761 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.235982 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.236192 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.236256 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.237193 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:22.237194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:22.237194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:22.237199 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:22.237216 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:22.237222 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:22.237279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.237373 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.237409 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:48:22.237684 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.232601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42413 (local address 127.0.98.1:33641)
0928 22:48:22.232804 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.232841 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.233347 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.233623 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.233628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.233637 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:22.234154 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.234159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.234861 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.234862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.235910 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.235913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.236279 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.236280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.236454 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.237034 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:22.237035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:22.237075 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:22.237104 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:22.237289 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:22.237294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:22.237298 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.237371 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.237477 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 22:48:22.238163 20229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42413
I20260928 22:48:22.238257 20229 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:22.238444 20229 heartbeater.cc:511] Master 127.0.98.62:42413 requested a full tablet report, sending...
I20260928 22:48:22.238950 20041 ts_manager.cc:194] Registered new tserver with Master: 3554b6cafd4846ca9d55987d654617c7 (127.0.98.1:33275)
I20260928 22:48:22.239639 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33641
W20260928 22:48:22.316522 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:22.316783 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:22.316816 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:22.318068 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:22.318120 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:22.318157 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 22:48:22.319653 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:22.319900 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:22.320189 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:48:22.325213 20232 server_base.cc:1034] running on GCE node
W20260928 22:48:22.325150 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.325192 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.325150 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.325613 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:22.325867 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:22.327016 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790635702327001 us; error 30 us; skew 500 ppm
I20260928 22:48:22.328233 20232 webserver.cc:460] Webserver started at http://127.0.98.2:34849/ using document root <none> and password file <none>
I20260928 22:48:22.328485 20232 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:22.328536 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:22.328658 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:22.329738 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data/instance:
uuid: "3fe53588cee54b45a3e003f796575271"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "13119981f152dfa8a5550fc7a98d7ab9"
server_key_iv: "f0d683f12df99dbd67e29b293475b3a3"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.330071 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3fe53588cee54b45a3e003f796575271"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "13119981f152dfa8a5550fc7a98d7ab9"
server_key_iv: "f0d683f12df99dbd67e29b293475b3a3"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.331362 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:22.332110 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.332284 20232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:22.332366 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal
uuid: "3fe53588cee54b45a3e003f796575271"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "13119981f152dfa8a5550fc7a98d7ab9"
server_key_iv: "f0d683f12df99dbd67e29b293475b3a3"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.332429 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:22.344067 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:22.344365 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:22.344496 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:22.344754 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:22.345089 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:22.345127 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.345155 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:22.345180 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.350478 20232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45035
I20260928 22:48:22.350541 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45035 every 8 connection(s)
I20260928 22:48:22.350835 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:22.351091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20232
I20260928 22:48:22.351210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:22.351414 392 external_mini_cluster.cc:1428] Setting key 393bb3abdb78f5828f7f25ed83a75093
I20260928 22:48:22.352397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:48:22.356860 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.351629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46959 (local address 127.0.98.62:42413)
0928 22:48:22.351686 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.351688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.351963 (+ 275us) server_negotiation.cc:407] Connection header received
0928 22:48:22.352496 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.352497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.352508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.352575 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:22.353511 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.354252 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.355293 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.355607 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.355687 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.356571 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:22.356573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:22.356573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:22.356579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:22.356599 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:22.356604 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:22.356662 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.356739 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.356769 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:48:22.357064 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.351647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42413 (local address 127.0.98.2:46959)
0928 22:48:22.351836 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.351871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.352387 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.352685 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.352689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.352701 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:22.353390 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.353397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.354385 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.354386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.355195 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.355197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.355723 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.355724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.355855 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.356427 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:22.356428 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:22.356459 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:22.356483 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:22.356661 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:22.356663 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:22.356666 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.356734 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.356845 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 22:48:22.357517 20360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42413
I20260928 22:48:22.357604 20360 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:22.357774 20360 heartbeater.cc:511] Master 127.0.98.62:42413 requested a full tablet report, sending...
I20260928 22:48:22.358155 20041 ts_manager.cc:194] Registered new tserver with Master: 3fe53588cee54b45a3e003f796575271 (127.0.98.2:45035)
I20260928 22:48:22.358494 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46959
W20260928 22:48:22.432065 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:22.432279 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:22.432308 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:22.433506 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:22.433548 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:22.433581 20363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 22:48:22.434991 20363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:22.435217 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:22.435491 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:22.440243 20369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:22.440280 20368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.440363 20363 server_base.cc:1034] running on GCE node
W20260928 22:48:22.440280 20371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:22.440678 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:22.440914 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:22.442054 20363 hybrid_clock.cc:648] HybridClock initialized: now 1790635702442039 us; error 29 us; skew 500 ppm
I20260928 22:48:22.443245 20363 webserver.cc:460] Webserver started at http://127.0.98.3:35035/ using document root <none> and password file <none>
I20260928 22:48:22.443457 20363 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:22.443523 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:22.443652 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:22.444697 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3f624d708334ac18e68813b3c30c255"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "52643933355d4ff5386763afffcb6a15"
server_key_iv: "3ab629b5ec9e99873305df0f01ca4aec"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.445072 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3f624d708334ac18e68813b3c30c255"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "52643933355d4ff5386763afffcb6a15"
server_key_iv: "3ab629b5ec9e99873305df0f01ca4aec"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.446280 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:22.446985 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.447124 20363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:22.447203 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal
uuid: "a3f624d708334ac18e68813b3c30c255"
format_stamp: "Formatted at 2026-09-28 22:48:22 on dist-test-slave-pxr1"
server_key: "52643933355d4ff5386763afffcb6a15"
server_key_iv: "3ab629b5ec9e99873305df0f01ca4aec"
server_key_version: "encryptionkey@0"
I20260928 22:48:22.447270 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:22.469226 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:22.469539 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:22.469664 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:22.469885 20363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:22.470216 20363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:22.470253 20363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.470283 20363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:22.470307 20363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:22.475598 20363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41777
I20260928 22:48:22.475658 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41777 every 8 connection(s)
I20260928 22:48:22.476008 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:22.477527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20363
I20260928 22:48:22.477624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:22.477788 392 external_mini_cluster.cc:1428] Setting key 784e13191f7765df124d4985d5e1403f
I20260928 22:48:22.481102 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.476711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57873 (local address 127.0.98.62:42413)
0928 22:48:22.476787 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.476790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.477004 (+ 214us) server_negotiation.cc:407] Connection header received
0928 22:48:22.477558 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.477560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.477573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.477625 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:22.478362 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.478857 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.479817 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.480017 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.480072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.480838 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:22.480839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:22.480839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:22.480845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:22.480860 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:22.480865 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:22.480911 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.480988 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.481019 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:22.481312 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.476692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42413 (local address 127.0.98.3:57873)
0928 22:48:22.476886 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.476920 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.477442 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.477697 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.477701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.477712 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:22.478268 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.478273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.478940 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.478941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.479730 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.479733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.480086 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.480087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.480218 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.480683 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:48:22.480683 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:22.480711 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:48:22.480766 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:22.480917 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:22.480923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:22.480927 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.480987 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.481093 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260928 22:48:22.481796 20491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42413
I20260928 22:48:22.481886 20491 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:22.482033 20491 heartbeater.cc:511] Master 127.0.98.62:42413 requested a full tablet report, sending...
I20260928 22:48:22.482393 20041 ts_manager.cc:194] Registered new tserver with Master: a3f624d708334ac18e68813b3c30c255 (127.0.98.3:41777)
I20260928 22:48:22.482745 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57873
I20260928 22:48:22.489434 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:22.493311 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.490678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.0.98.62:42413)
0928 22:48:22.490732 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.490734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.490873 (+ 139us) server_negotiation.cc:407] Connection header received
0928 22:48:22.490917 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.490918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.490926 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.490978 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:22.491104 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.491597 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.492299 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.492474 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.492515 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.493127 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:22.493128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:22.493128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:22.493132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:22.493145 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:22.493148 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:22.493177 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.493213 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.493234 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:48:22.494815 20041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:22.496361 20041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:22.502265 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45035 (local address 127.0.0.1:56966)
0928 22:48:22.498698 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.498741 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.498797 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.499059 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.499062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.499064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:22.499132 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.499135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500167 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.500167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.501183 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.501186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.501976 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.501976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.502049 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502064 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.502132 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.502185 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":63,"threads_started":1}
I20260928 22:48:22.502272 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.98.2:45035)
0928 22:48:22.498925 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.498929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.498937 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:22.498968 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.498969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.498981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.499046 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:22.499205 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500107 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.501266 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.501909 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.501978 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502067 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.502147 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.502185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":81,"threads_started":1}
I20260928 22:48:22.502885 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41777 (local address 127.0.0.1:39958)
0928 22:48:22.499141 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.499151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.499195 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.499366 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.499367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.499368 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:22.499402 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.499403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500644 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.500645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.501799 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.501802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.502648 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.502649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.502708 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502712 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.502777 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.502816 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":56,"threads_started":1}
I20260928 22:48:22.502913 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.0.98.3:41777)
0928 22:48:22.498855 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.498860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.499246 (+ 386us) server_negotiation.cc:407] Connection header received
0928 22:48:22.499287 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.499289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.499304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.499371 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:22.499473 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500558 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.501890 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.502575 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.502636 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502713 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.502792 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.502828 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260928 22:48:22.503284 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33275 (local address 127.0.0.1:51996)
0928 22:48:22.498701 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.498736 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.498793 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.499642 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.499643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.499645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:22.499685 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.499686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500967 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.500968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.502047 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.502048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.502899 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.502899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.502954 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502957 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.503186 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.503225 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":84,"threads_started":1}
I20260928 22:48:22.503453 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.498495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.1:33275)
0928 22:48:22.499500 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.499505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.499514 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:22.499547 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.499549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.499562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.499634 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:22.499751 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.500889 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.502127 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.502721 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.502775 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.502867 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.503214 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.503250 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":93,"threads_started":1}
I20260928 22:48:22.503348 20294 tablet_service.cc:1467] Processing CreateTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 (DEFAULT_TABLE table=test-table [id=a94942ffbb6544fda95838151e2e158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:22.503744 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:22.503746 20425 tablet_service.cc:1467] Processing CreateTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 (DEFAULT_TABLE table=test-table [id=a94942ffbb6544fda95838151e2e158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:22.503968 20163 tablet_service.cc:1467] Processing CreateTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 (DEFAULT_TABLE table=test-table [id=a94942ffbb6544fda95838151e2e158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:22.504055 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:22.504196 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:22.506872 20505 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: Bootstrap starting.
I20260928 22:48:22.507244 20507 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: Bootstrap starting.
I20260928 22:48:22.507454 20506 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Bootstrap starting.
I20260928 22:48:22.507550 20505 tablet_bootstrap.cc:654] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:22.507853 20507 tablet_bootstrap.cc:654] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:22.507869 20505 log.cc:824] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:22.507994 20506 tablet_bootstrap.cc:654] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:22.508154 20507 log.cc:824] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:22.508286 20506 log.cc:824] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:22.508612 20505 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: No bootstrap required, opened a new log
I20260928 22:48:22.508685 20505 ts_tablet_manager.cc:1397] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:22.508868 20507 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: No bootstrap required, opened a new log
I20260928 22:48:22.508930 20507 ts_tablet_manager.cc:1397] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:22.508987 20506 tablet_bootstrap.cc:492] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: No bootstrap required, opened a new log
I20260928 22:48:22.509054 20506 ts_tablet_manager.cc:1397] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:22.510941 20506 raft_consensus.cc:348] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.511080 20506 raft_consensus.cc:374] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:22.511112 20506 raft_consensus.cc:729] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f624d708334ac18e68813b3c30c255, State: Initialized, Role: FOLLOWER
I20260928 22:48:22.511250 20506 consensus_queue.cc:260] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.511320 20507 raft_consensus.cc:348] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.511502 20507 raft_consensus.cc:374] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:22.511418 20505 raft_consensus.cc:348] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.511543 20505 raft_consensus.cc:374] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:22.511543 20507 raft_consensus.cc:729] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe53588cee54b45a3e003f796575271, State: Initialized, Role: FOLLOWER
I20260928 22:48:22.511540 20506 ts_tablet_manager.cc:1428] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:48:22.511581 20505 raft_consensus.cc:729] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3554b6cafd4846ca9d55987d654617c7, State: Initialized, Role: FOLLOWER
I20260928 22:48:22.511675 20491 heartbeater.cc:503] Master 127.0.98.62:42413 was elected leader, sending a full tablet report...
I20260928 22:48:22.511685 20505 consensus_queue.cc:260] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.511938 20505 ts_tablet_manager.cc:1428] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:48:22.511988 20507 consensus_queue.cc:260] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.512048 20229 heartbeater.cc:503] Master 127.0.98.62:42413 was elected leader, sending a full tablet report...
I20260928 22:48:22.512231 20507 ts_tablet_manager.cc:1428] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:48:22.512276 20360 heartbeater.cc:503] Master 127.0.98.62:42413 was elected leader, sending a full tablet report...
I20260928 22:48:22.548341 20512 raft_consensus.cc:482] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:22.548470 20512 raft_consensus.cc:504] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.548810 20512 leader_election.cc:288] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fe53588cee54b45a3e003f796575271 (127.0.98.2:45035), a3f624d708334ac18e68813b3c30c255 (127.0.98.3:41777)
I20260928 22:48:22.551592 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.548999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45035 (local address 127.0.98.1:33281)
0928 22:48:22.549053 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.549067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.549124 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.549281 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.549282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.549303 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:22.549364 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.549366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.550001 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.550002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.550823 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.550826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.551363 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.551364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.551423 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.551426 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.551468 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.551515 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:48:22.551592 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.549096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33281 (local address 127.0.98.2:45035)
0928 22:48:22.549145 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.549147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.549158 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:48:22.549188 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.549189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.549198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.549249 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:22.549431 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.549928 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.550910 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.551272 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.551347 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.551358 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.551483 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.551521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:48:22.551841 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.549208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48595 (local address 127.0.98.3:41777)
0928 22:48:22.549250 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.549252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.549353 (+ 101us) server_negotiation.cc:407] Connection header received
0928 22:48:22.549417 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.549417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.549426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.549473 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:22.549619 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.550078 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.551108 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.551481 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.551541 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.551551 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.551721 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.551771 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:22.551934 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4fe7a2c3ad45deb6e0eee7fb1f5620" candidate_uuid: "3554b6cafd4846ca9d55987d654617c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe53588cee54b45a3e003f796575271" is_pre_election: true
I20260928 22:48:22.552068 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.549105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41777 (local address 127.0.98.1:48595)
0928 22:48:22.549305 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.549314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:22.549365 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:22.549495 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:22.549496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:22.549497 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:22.549563 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.549564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.550149 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.550150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.551030 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:22.551032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.551565 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.551566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:22.551633 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.551636 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:48:22.551948 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 22:48:22.551991 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":120,"threads_started":1}
I20260928 22:48:22.552102 20314 raft_consensus.cc:2454] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3554b6cafd4846ca9d55987d654617c7 in term 0.
I20260928 22:48:22.552320 20116 leader_election.cc:302] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3554b6cafd4846ca9d55987d654617c7, 3fe53588cee54b45a3e003f796575271; no voters:
I20260928 22:48:22.552383 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4fe7a2c3ad45deb6e0eee7fb1f5620" candidate_uuid: "3554b6cafd4846ca9d55987d654617c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f624d708334ac18e68813b3c30c255" is_pre_election: true
I20260928 22:48:22.552441 20512 raft_consensus.cc:2781] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:22.552495 20512 raft_consensus.cc:482] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:22.552520 20512 raft_consensus.cc:3037] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:22.552526 20445 raft_consensus.cc:2454] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3554b6cafd4846ca9d55987d654617c7 in term 0.
I20260928 22:48:22.553297 20512 raft_consensus.cc:504] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.553423 20512 leader_election.cc:288] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [CANDIDATE]: Term 1 election: Requested vote from peers 3fe53588cee54b45a3e003f796575271 (127.0.98.2:45035), a3f624d708334ac18e68813b3c30c255 (127.0.98.3:41777)
I20260928 22:48:22.553613 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4fe7a2c3ad45deb6e0eee7fb1f5620" candidate_uuid: "3554b6cafd4846ca9d55987d654617c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe53588cee54b45a3e003f796575271"
I20260928 22:48:22.553635 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4fe7a2c3ad45deb6e0eee7fb1f5620" candidate_uuid: "3554b6cafd4846ca9d55987d654617c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f624d708334ac18e68813b3c30c255"
I20260928 22:48:22.553705 20445 raft_consensus.cc:3037] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:22.553706 20314 raft_consensus.cc:3037] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:22.554579 20314 raft_consensus.cc:2454] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3554b6cafd4846ca9d55987d654617c7 in term 1.
I20260928 22:48:22.554579 20445 raft_consensus.cc:2454] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3554b6cafd4846ca9d55987d654617c7 in term 1.
I20260928 22:48:22.554765 20116 leader_election.cc:302] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3554b6cafd4846ca9d55987d654617c7, 3fe53588cee54b45a3e003f796575271; no voters:
I20260928 22:48:22.554858 20512 raft_consensus.cc:2781] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:22.555011 20512 raft_consensus.cc:686] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3554b6cafd4846ca9d55987d654617c7, State: Running, Role: LEADER
I20260928 22:48:22.555111 20512 consensus_queue.cc:237] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } }
I20260928 22:48:22.556016 20039 catalog_manager.cc:5495] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3554b6cafd4846ca9d55987d654617c7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3554b6cafd4846ca9d55987d654617c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3554b6cafd4846ca9d55987d654617c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20260928 22:48:22.575313 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.573270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.0.98.1:33275)
0928 22:48:22.573327 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:22.573329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:22.573339 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:48:22.573376 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:22.573377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:22.573395 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:22.573451 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:22.573570 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.574052 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.574734 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:22.574910 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:22.574951 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:22.575050 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:22.575147 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:22.575204 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:22.575216 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 22:48:22.575234 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:48:22.577117 20445 raft_consensus.cc:1261] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 1 FOLLOWER]: Refusing update from remote peer 3554b6cafd4846ca9d55987d654617c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:22.577138 20314 raft_consensus.cc:1261] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 1 FOLLOWER]: Refusing update from remote peer 3554b6cafd4846ca9d55987d654617c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:22.577499 20515 consensus_queue.cc:1035] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe53588cee54b45a3e003f796575271" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:22.577637 20512 consensus_queue.cc:1035] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3554b6cafd4846ca9d55987d654617c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f624d708334ac18e68813b3c30c255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:22.579895 20518 mvcc.cc:205] Tried to move back new op lower bound from 7334443837753286656 to 7334443837666410496. Current Snapshot: MvccSnapshot[applied={T|T < 7334443837753286656}]
I20260928 22:48:22.580420 20520 mvcc.cc:205] Tried to move back new op lower bound from 7334443837753286656 to 7334443837666410496. Current Snapshot: MvccSnapshot[applied={T|T < 7334443837753286656}]
I20260928 22:48:22.580665 20519 mvcc.cc:205] Tried to move back new op lower bound from 7334443837753286656 to 7334443837666410496. Current Snapshot: MvccSnapshot[applied={T|T < 7334443837753286656}]
I20260928 22:48:22.582326 20039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:22.582415 20039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:22.583123 20039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64035475ff62477eb3f08ea871c1fb1e: Sending DeleteTablet for 3 replicas of tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620
I20260928 22:48:22.583521 20294 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:22 UTC) from {username='slave'} at 127.0.0.1:56966
I20260928 22:48:22.583503 20425 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:22 UTC) from {username='slave'} at 127.0.0.1:39958
I20260928 22:48:22.583524 20163 tablet_service.cc:1514] Processing DeleteTablet for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:22 UTC) from {username='slave'} at 127.0.0.1:51996
I20260928 22:48:22.583981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20101
I20260928 22:48:22.583921 20528 tablet_replica.cc:330] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: stopping tablet replica
I20260928 22:48:22.588163 20528 raft_consensus.cc:2229] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:22.588303 20528 raft_consensus.cc:2258] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:48:22.588433 20026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33275 (error 108)
I20260928 22:48:22.588565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20232
I20260928 22:48:22.588574 20529 tablet_replica.cc:330] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271: stopping tablet replica
I20260928 22:48:22.588672 20529 raft_consensus.cc:2229] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:22.588789 20529 raft_consensus.cc:2258] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P 3fe53588cee54b45a3e003f796575271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:22.589170 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.588983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33275 (local address 127.0.0.1:52014)
0928 22:48:22.589083 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.589135 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 22:48:22.589306 20026 catalog_manager.cc:4584] TS 3554b6cafd4846ca9d55987d654617c7 (127.0.98.1:33275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33275: connect: Connection refused (error 111)
I20260928 22:48:22.590265 20528 ts_tablet_manager.cc:1905] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:22.591933 20528 ts_tablet_manager.cc:1918] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:22.592013 20528 log.cc:1196] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Deleting WAL directory at /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635468897675-392-0/minicluster-data/ts-2/wal/wals/ab4fe7a2c3ad45deb6e0eee7fb1f5620
I20260928 22:48:22.592245 20528 ts_tablet_manager.cc:1939] T ab4fe7a2c3ad45deb6e0eee7fb1f5620 P a3f624d708334ac18e68813b3c30c255: Deleting consensus metadata
I20260928 22:48:22.592651 20025 catalog_manager.cc:4841] TS a3f624d708334ac18e68813b3c30c255 (127.0.98.3:41777): tablet ab4fe7a2c3ad45deb6e0eee7fb1f5620 (table test-table [id=a94942ffbb6544fda95838151e2e158e]) successfully deleted
I20260928 22:48:22.594215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20363
I20260928 22:48:22.594254 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:22.594149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45035 (local address 127.0.0.1:56982)
0928 22:48:22.594196 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:22.594224 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 22:48:22.598802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20010
2026-09-28T22:48:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (646 ms)
[----------] 6 tests from EncryptionPolicyTest (8102 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:22 dist-test-slave-pxr1 krb5kdc[20534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:22 dist-test-slave-pxr1 krb5kdc[20534](info): set up 2 sockets
Sep 28 22:48:22 dist-test-slave-pxr1 krb5kdc[20534](info): commencing operation
krb5kdc: starting...
W20260928 22:48:23.553783 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:23.675980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:23.755141 20547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:23.755358 20547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:23.755389 20547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:23.756585 20547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:23.756629 20547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:23.756649 20547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:23.756664 20547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:23.756678 20547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:23.758188 20547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:23.758442 20547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:23.758646 20547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:23.763211 20555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:23.763206 20553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:23.763298 20547 server_base.cc:1034] running on GCE node
W20260928 22:48:23.763206 20552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:23.763631 20547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:23.763911 20547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:23.765043 20547 hybrid_clock.cc:648] HybridClock initialized: now 1790635703765032 us; error 28 us; skew 500 ppm
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:23.767280 20547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:23.767653 20547 webserver.cc:460] Webserver started at http://127.0.0.1:45195/ using document root <none> and password file <none>
I20260928 22:48:23.767838 20547 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:23.767876 20547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:23.767966 20547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:23.768996 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "d0c01501109e4485af81a4f818c30236"
format_stamp: "Formatted at 2026-09-28 22:48:23 on dist-test-slave-pxr1"
server_key: "3605b3c16a6778ca1a1f57d629bd3782"
server_key_iv: "293635be4424607aedab56599a5167cd"
server_key_version: "encryptionkey@0"
I20260928 22:48:23.769280 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0c01501109e4485af81a4f818c30236"
format_stamp: "Formatted at 2026-09-28 22:48:23 on dist-test-slave-pxr1"
server_key: "3605b3c16a6778ca1a1f57d629bd3782"
server_key_iv: "293635be4424607aedab56599a5167cd"
server_key_version: "encryptionkey@0"
I20260928 22:48:23.770462 20547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:23.771078 20562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:23.771230 20547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:23.771312 20547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "d0c01501109e4485af81a4f818c30236"
format_stamp: "Formatted at 2026-09-28 22:48:23 on dist-test-slave-pxr1"
server_key: "3605b3c16a6778ca1a1f57d629bd3782"
server_key_iv: "293635be4424607aedab56599a5167cd"
server_key_version: "encryptionkey@0"
I20260928 22:48:23.771376 20547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:23.786612 20547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:23.788367 20547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:23.788523 20547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:23.792264 20547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45985
I20260928 22:48:23.792309 20614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45985 every 8 connection(s)
I20260928 22:48:23.792827 20547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:23.793475 20615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:23.795713 20615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: Bootstrap starting.
I20260928 22:48:23.796265 20615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:23.796489 20615 log.cc:824] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:23.797183 20615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: No bootstrap required, opened a new log
I20260928 22:48:23.798719 20615 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260928 22:48:23.798933 20615 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:23.798978 20615 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c01501109e4485af81a4f818c30236, State: Initialized, Role: FOLLOWER
I20260928 22:48:23.799096 20615 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260928 22:48:23.799149 20615 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:23.799168 20615 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:23.799197 20615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:23.800060 20615 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260928 22:48:23.800144 20615 leader_election.cc:302] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c01501109e4485af81a4f818c30236; no voters:
I20260928 22:48:23.800272 20615 leader_election.cc:288] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:23.800541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20547
I20260928 22:48:23.800655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:23.800881 392 external_mini_cluster.cc:1428] Setting key 1c2f99eb404d52e030357dfc03971da8
I20260928 22:48:23.800333 20618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:23.800479 20615 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:23.801105 20618 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [term 1 LEADER]: Becoming Leader. State: Replica: d0c01501109e4485af81a4f818c30236, State: Running, Role: LEADER
I20260928 22:48:23.801184 20618 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } }
I20260928 22:48:23.801553 20620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c01501109e4485af81a4f818c30236. Latest consensus state: current_term: 1 leader_uuid: "d0c01501109e4485af81a4f818c30236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } } }
I20260928 22:48:23.801728 20620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:23.801708 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c01501109e4485af81a4f818c30236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c01501109e4485af81a4f818c30236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45985 } } }
I20260928 22:48:23.801862 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236 [sys.catalog]: This master's current role is: LEADER
W20260928 22:48:23.803010 20635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:23.803109 20635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:23.803213 20630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:23.803387 20630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:23.804633 20630 catalog_manager.cc:1347] Generated new cluster ID: eb64410266a14766823404c9dfbb1c45
I20260928 22:48:23.804678 20630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:48:23.812704 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:23.801348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.0.0.1:45985)
0928 22:48:23.801542 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:23.801552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:23.801749 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:48:23.801964 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:23.801973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:23.802034 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:23.802148 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:23.803189 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:23.804590 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:23.805695 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:23.806303 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:23.806426 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:23.809062 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:23.809067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:23.809067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:23.809080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:23.811038 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:23.811464 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:23.811468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:23.811468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:23.811595 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:23.811802 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:23.811803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:23.811803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:23.812132 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:23.812300 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:23.812380 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:48:23.812482 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260928 22:48:23.813699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:23.822576 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:23.815052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41177 (local address 127.0.0.1:45985)
0928 22:48:23.815083 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:23.815084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:23.815200 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:48:23.815256 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:23.815257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:23.815268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:23.815316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:23.815871 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:23.816672 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:23.817658 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:23.818008 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:23.818065 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:23.819344 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:23.819345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:23.819346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:23.819353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:23.821433 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:23.821830 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:23.821831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:23.821831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:23.821890 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:23.822074 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:23.822074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:23.822074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:23.822205 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:23.822304 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:23.822413 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:48:23.822478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:23.826745 20630 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:23.827245 20630 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:23.836560 20630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c01501109e4485af81a4f818c30236: Generated new TSK 0
I20260928 22:48:23.836901 20630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:23.847168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20547
W20260928 22:48:23.851325 20532 connection.cc:535] client connection to 127.0.0.1:45985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45985): Connection reset by peer (error 104)
2026-09-28T22:48:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20646](info): set up 2 sockets
Sep 28 22:48:23 dist-test-slave-pxr1 krb5kdc[20646](info): commencing operation
krb5kdc: starting...
W20260928 22:48:24.800403 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:24 dist-test-slave-pxr1 krb5kdc[20646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:24.911495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:24.991291 20659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:24.991505 20659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:24.991537 20659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:24.992765 20659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:24.992810 20659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:24.992826 20659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:24.992842 20659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:24.992857 20659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:24.994360 20659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:24.994611 20659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:24.994823 20659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:48:24.999682 20659 server_base.cc:1034] running on GCE node
W20260928 22:48:24.999647 20664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:24.999663 20667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:24.999663 20665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:25.000079 20659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:25.000353 20659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:25.001489 20659 hybrid_clock.cc:648] HybridClock initialized: now 1790635705001474 us; error 33 us; skew 500 ppm
Sep 28 22:48:24 dist-test-slave-pxr1 krb5kdc[20646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:25.003902 20659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:25.004302 20659 webserver.cc:460] Webserver started at http://127.0.0.1:35783/ using document root <none> and password file <none>
I20260928 22:48:25.004494 20659 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:25.004542 20659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:25.004663 20659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:25.005755 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "5f4126f3ea14437d92b13eab24f6c11f"
format_stamp: "Formatted at 2026-09-28 22:48:25 on dist-test-slave-pxr1"
server_key: "bf0a0c66c5e80e75ad47be9ef4817b87"
server_key_iv: "723ea4032c509e48d35a3af14f4bba65"
server_key_version: "encryptionkey@0"
I20260928 22:48:25.006096 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f4126f3ea14437d92b13eab24f6c11f"
format_stamp: "Formatted at 2026-09-28 22:48:25 on dist-test-slave-pxr1"
server_key: "bf0a0c66c5e80e75ad47be9ef4817b87"
server_key_iv: "723ea4032c509e48d35a3af14f4bba65"
server_key_version: "encryptionkey@0"
I20260928 22:48:25.007385 20659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:25.008028 20674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:25.008173 20659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:25.008270 20659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "5f4126f3ea14437d92b13eab24f6c11f"
format_stamp: "Formatted at 2026-09-28 22:48:25 on dist-test-slave-pxr1"
server_key: "bf0a0c66c5e80e75ad47be9ef4817b87"
server_key_iv: "723ea4032c509e48d35a3af14f4bba65"
server_key_version: "encryptionkey@0"
I20260928 22:48:25.008345 20659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:25.036031 20659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:25.038362 20659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:25.038514 20659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:25.042155 20659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45487
I20260928 22:48:25.042200 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45487 every 8 connection(s)
I20260928 22:48:25.042533 20659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:25.043399 20727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:25.045691 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f: Bootstrap starting.
I20260928 22:48:25.046078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20659
I20260928 22:48:25.046191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:25.046386 392 external_mini_cluster.cc:1428] Setting key 9520264cefc2245f876d94b4deab51ad
I20260928 22:48:25.046388 20727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:25.046674 20727 log.cc:824] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:25.047519 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f: No bootstrap required, opened a new log
I20260928 22:48:25.050077 20727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } }
I20260928 22:48:25.050227 20727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:25.050256 20727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4126f3ea14437d92b13eab24f6c11f, State: Initialized, Role: FOLLOWER
I20260928 22:48:25.050361 20727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } }
I20260928 22:48:25.050431 20727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:25.050454 20727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:25.050498 20727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:25.051277 20727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } }
I20260928 22:48:25.051388 20727 leader_election.cc:302] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f4126f3ea14437d92b13eab24f6c11f; no voters:
I20260928 22:48:25.051550 20727 leader_election.cc:288] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:25.051575 20732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:25.051685 20732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [term 1 LEADER]: Becoming Leader. State: Replica: 5f4126f3ea14437d92b13eab24f6c11f, State: Running, Role: LEADER
I20260928 22:48:25.051786 20732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } }
I20260928 22:48:25.051820 20727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:25.052119 20734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f4126f3ea14437d92b13eab24f6c11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } } }
I20260928 22:48:25.052256 20734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:25.052815 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:25.046948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59125 (local address 127.0.0.1:45487)
0928 22:48:25.047129 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:25.047137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:25.047163 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:48:25.047760 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:25.047769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:25.047835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:25.047988 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:25.048204 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:25.049643 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:25.050545 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:25.051179 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:25.051296 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:25.052038 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:25.052043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:25.052044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:25.052282 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:25.052325 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:25.052341 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:25.052426 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:25.052521 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:48:25.052550 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 22:48:25.053099 20742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:25.052471 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f4126f3ea14437d92b13eab24f6c11f. Latest consensus state: current_term: 1 leader_uuid: "5f4126f3ea14437d92b13eab24f6c11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4126f3ea14437d92b13eab24f6c11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45487 } } }
I20260928 22:48:25.053428 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:25.053810 20742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:25.053833 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:25.055477 20742 catalog_manager.cc:1347] Generated new cluster ID: 59af347c093045939aef517043fc2264
I20260928 22:48:25.055526 20742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:25.059011 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:25.055344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44635 (local address 127.0.0.1:45487)
0928 22:48:25.055379 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:25.055381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:25.055590 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:48:25.055682 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:25.055683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:25.055694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:25.055747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:25.055902 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:25.056635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:25.057689 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:25.058020 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:25.058077 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:25.058770 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:25.058772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:25.058772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:25.058777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:25.058792 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:25.058796 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:25.058846 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:25.058926 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:48:25.058936 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:25.065544 20742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:25.066080 20742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:25.077921 20742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f4126f3ea14437d92b13eab24f6c11f: Generated new TSK 0
I20260928 22:48:25.078121 20742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:25.082084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20659
W20260928 22:48:25.086653 20644 connection.cc:535] client connection to 127.0.0.1:45487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45487): Connection reset by peer (error 104)
2026-09-28T22:48:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:25 dist-test-slave-pxr1 krb5kdc[20758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:25 dist-test-slave-pxr1 krb5kdc[20758](info): set up 2 sockets
Sep 28 22:48:25 dist-test-slave-pxr1 krb5kdc[20758](info): commencing operation
krb5kdc: starting...
W20260928 22:48:26.040474 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:26 dist-test-slave-pxr1 krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:26.151458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:26.230939 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:26.231192 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:26.231218 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:26.232470 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:26.232522 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:26.232535 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:26.232547 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:26.232558 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:26.234154 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:26.234385 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:26.234576 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:26.239327 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:26.239319 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:26.239318 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:26.239352 20771 server_base.cc:1034] running on GCE node
I20260928 22:48:26.239683 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:26.239946 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:26.241106 20771 hybrid_clock.cc:648] HybridClock initialized: now 1790635706241085 us; error 34 us; skew 500 ppm
Sep 28 22:48:26 dist-test-slave-pxr1 krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:26.243498 20771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:26.243952 20771 webserver.cc:460] Webserver started at http://127.0.0.1:33487/ using document root <none> and password file <none>
I20260928 22:48:26.244167 20771 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:26.244215 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:26.244345 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:26.245435 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "051d7921e0ca47eabb8e710a2606bfe9"
format_stamp: "Formatted at 2026-09-28 22:48:26 on dist-test-slave-pxr1"
server_key: "e195e61e017a36017ca9e2df6785d577"
server_key_iv: "8fe293e3217ecd911934c8b94fb1e487"
server_key_version: "encryptionkey@0"
I20260928 22:48:26.245757 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "051d7921e0ca47eabb8e710a2606bfe9"
format_stamp: "Formatted at 2026-09-28 22:48:26 on dist-test-slave-pxr1"
server_key: "e195e61e017a36017ca9e2df6785d577"
server_key_iv: "8fe293e3217ecd911934c8b94fb1e487"
server_key_version: "encryptionkey@0"
I20260928 22:48:26.246996 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:26.247655 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:26.247812 20771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:26.247896 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "051d7921e0ca47eabb8e710a2606bfe9"
format_stamp: "Formatted at 2026-09-28 22:48:26 on dist-test-slave-pxr1"
server_key: "e195e61e017a36017ca9e2df6785d577"
server_key_iv: "8fe293e3217ecd911934c8b94fb1e487"
server_key_version: "encryptionkey@0"
I20260928 22:48:26.247958 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:26.251215 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:26.253082 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:26.253208 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:26.256927 20771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40479
I20260928 22:48:26.256958 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40479 every 8 connection(s)
I20260928 22:48:26.257264 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:26.258193 20839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:26.260394 20839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9: Bootstrap starting.
I20260928 22:48:26.260972 20839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:26.261238 20839 log.cc:824] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:26.261854 20839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9: No bootstrap required, opened a new log
I20260928 22:48:26.263443 20839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260928 22:48:26.263594 20839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:26.263616 20839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051d7921e0ca47eabb8e710a2606bfe9, State: Initialized, Role: FOLLOWER
I20260928 22:48:26.263756 20839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260928 22:48:26.263814 20839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:26.263845 20839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:26.263890 20839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:26.264460 20839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260928 22:48:26.264550 20839 leader_election.cc:302] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 051d7921e0ca47eabb8e710a2606bfe9; no voters:
I20260928 22:48:26.264704 20839 leader_election.cc:288] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:26.264811 20842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:26.264983 20839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:26.265163 20842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 051d7921e0ca47eabb8e710a2606bfe9, State: Running, Role: LEADER
I20260928 22:48:26.265257 20842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260928 22:48:26.265582 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "051d7921e0ca47eabb8e710a2606bfe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260928 22:48:26.265605 20844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 051d7921e0ca47eabb8e710a2606bfe9. Latest consensus state: current_term: 1 leader_uuid: "051d7921e0ca47eabb8e710a2606bfe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "051d7921e0ca47eabb8e710a2606bfe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260928 22:48:26.265719 20844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:26.266000 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:26.266057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20771
I20260928 22:48:26.266173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:26.266335 392 external_mini_cluster.cc:1428] Setting key cbbfcc342b501c2b5683c8f54dafff5d
I20260928 22:48:26.266711 20851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:26.267251 20851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:26.268749 20851 catalog_manager.cc:1347] Generated new cluster ID: 323899269b814e7780639f5769e25728
I20260928 22:48:26.268801 20851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:26.274407 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:26.266955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33963 (local address 127.0.0.1:40479)
0928 22:48:26.272878 (+ 5923us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:26.272886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:26.272920 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:48:26.273039 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:26.273044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:26.273069 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:26.273220 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:26.273871 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:26.273875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:26.273875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:26.273879 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:26.273912 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:26.273924 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:26.273976 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:26.274040 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 22:48:26.274083 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5855,"thread_start_us":66,"threads_started":1}
I20260928 22:48:26.275430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:26.281965 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:26.276807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53579 (local address 127.0.0.1:40479)
0928 22:48:26.276844 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:26.276846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:26.281012 (+ 4166us) server_negotiation.cc:407] Connection header received
0928 22:48:26.281082 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:26.281083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:26.281089 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:26.281157 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:26.281747 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:26.281749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:26.281749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:26.281754 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:26.281770 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:26.281774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:26.281822 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:26.281870 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 22:48:26.281886 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:48:26.290310 20851 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:26.290495 20851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:26.298626 20851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 051d7921e0ca47eabb8e710a2606bfe9: Generated new TSK 0
I20260928 22:48:26.298823 20851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:26.306723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20771
2026-09-28T22:48:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:26 dist-test-slave-pxr1 krb5kdc[20870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:26 dist-test-slave-pxr1 krb5kdc[20870](info): set up 2 sockets
Sep 28 22:48:26 dist-test-slave-pxr1 krb5kdc[20870](info): commencing operation
krb5kdc: starting...
W20260928 22:48:27.256582 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:27.366583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:27.445716 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:27.445940 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:27.445971 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:27.447188 20883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:27.447235 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:27.447254 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:27.447270 20883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:27.447285 20883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:27.448827 20883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:27.449074 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:27.449280 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:27.453931 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:27.453931 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:27.453989 20883 server_base.cc:1034] running on GCE node
W20260928 22:48:27.453958 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:27.454334 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:27.454600 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:27.455734 20883 hybrid_clock.cc:648] HybridClock initialized: now 1790635707455722 us; error 35 us; skew 500 ppm
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:27.458069 20883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:27.458478 20883 webserver.cc:460] Webserver started at http://127.0.0.1:45287/ using document root <none> and password file <none>
I20260928 22:48:27.458671 20883 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:27.458719 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:27.458837 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:27.459873 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f"
format_stamp: "Formatted at 2026-09-28 22:48:27 on dist-test-slave-pxr1"
server_key: "037c32c8316b2b44db69420acec03fd2"
server_key_iv: "b4508878837d5d1795251ce111035f71"
server_key_version: "encryptionkey@0"
I20260928 22:48:27.460198 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f"
format_stamp: "Formatted at 2026-09-28 22:48:27 on dist-test-slave-pxr1"
server_key: "037c32c8316b2b44db69420acec03fd2"
server_key_iv: "b4508878837d5d1795251ce111035f71"
server_key_version: "encryptionkey@0"
I20260928 22:48:27.461488 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:48:27.462232 20898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:27.462396 20883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:27.462477 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f"
format_stamp: "Formatted at 2026-09-28 22:48:27 on dist-test-slave-pxr1"
server_key: "037c32c8316b2b44db69420acec03fd2"
server_key_iv: "b4508878837d5d1795251ce111035f71"
server_key_version: "encryptionkey@0"
I20260928 22:48:27.462536 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:27.490871 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:27.492669 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:27.492828 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:27.496475 20883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43885
I20260928 22:48:27.496539 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43885 every 8 connection(s)
I20260928 22:48:27.496860 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:27.497669 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:27.499891 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f: Bootstrap starting.
I20260928 22:48:27.500439 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:27.500684 20951 log.cc:824] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:27.501144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20883
I20260928 22:48:27.501266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:27.501435 392 external_mini_cluster.cc:1428] Setting key 295618e21b41016ef1436820e4ea15f8
I20260928 22:48:27.501464 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f: No bootstrap required, opened a new log
I20260928 22:48:27.503634 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260928 22:48:27.503782 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:27.503803 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fab7b934e6d4f39a01bffdda8d2bc4f, State: Initialized, Role: FOLLOWER
I20260928 22:48:27.503911 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260928 22:48:27.503974 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:27.503996 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:27.504038 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:27.504573 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260928 22:48:27.504658 20951 leader_election.cc:302] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fab7b934e6d4f39a01bffdda8d2bc4f; no voters:
I20260928 22:48:27.504837 20951 leader_election.cc:288] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:27.504876 20956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:27.505024 20956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 6fab7b934e6d4f39a01bffdda8d2bc4f, State: Running, Role: LEADER
I20260928 22:48:27.505093 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:27.505152 20956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260928 22:48:27.505493 20957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } } }
I20260928 22:48:27.505569 20957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:27.505795 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fab7b934e6d4f39a01bffdda8d2bc4f. Latest consensus state: current_term: 1 leader_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab7b934e6d4f39a01bffdda8d2bc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } } }
I20260928 22:48:27.505903 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:27.506054 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:27.506557 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:27.508129 20963 catalog_manager.cc:1347] Generated new cluster ID: 45303f71b0a549dc9ff18e97a025d7a7
I20260928 22:48:27.508183 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:48:27.512579 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:27.501911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42703 (local address 127.0.0.1:43885)
0928 22:48:27.502061 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:27.502070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:27.502105 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:48:27.502703 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:27.502710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:27.502756 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:27.502866 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:27.503616 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:27.504908 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:27.505955 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:27.506553 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:27.506659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:27.509404 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:27.509409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:27.509409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:27.509420 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:27.510928 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:27.511266 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:27.511269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:27.511270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:27.511350 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:27.511520 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:27.511520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:27.511521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:27.511748 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:27.512260 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:27.512332 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:48:27.512437 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 22:48:27.513568 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:27.516939 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:27.517606 20963 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:27.520370 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:27.514798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55321 (local address 127.0.0.1:43885)
0928 22:48:27.514832 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:27.514834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:27.514931 (+ 97us) server_negotiation.cc:407] Connection header received
0928 22:48:27.514985 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:27.514985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:27.514995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:27.515040 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:27.515510 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:27.515990 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:27.516693 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:27.517071 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:27.517136 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:27.518213 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:27.518214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:27.518215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:27.518222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:27.519421 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:27.519711 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:27.519712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:27.519712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:27.519763 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:27.519918 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:27.519919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:27.519919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:27.520019 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:27.520097 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:27.520203 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 22:48:27.520286 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:48:27.534284 20963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fab7b934e6d4f39a01bffdda8d2bc4f: Generated new TSK 0
I20260928 22:48:27.534503 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:27.541742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20883
2026-09-28T22:48:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20982](info): set up 2 sockets
Sep 28 22:48:27 dist-test-slave-pxr1 krb5kdc[20982](info): commencing operation
krb5kdc: starting...
W20260928 22:48:28.480491 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:28 dist-test-slave-pxr1 krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:28.593232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:28.672910 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:28.673131 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:28.673154 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:28.674333 20995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:28.674368 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:28.674381 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:28.674392 20995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:28.674402 20995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:28.675892 20995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:28.676100 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:28.676281 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:28.681130 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:28.681186 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:28.681131 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:28.681371 20995 server_base.cc:1034] running on GCE node
I20260928 22:48:28.681597 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:28.681847 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:28.682999 20995 hybrid_clock.cc:648] HybridClock initialized: now 1790635708682984 us; error 30 us; skew 500 ppm
Sep 28 22:48:28 dist-test-slave-pxr1 krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:28.689529 20995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:28.689990 20995 webserver.cc:460] Webserver started at http://127.0.0.1:43597/ using document root <none> and password file <none>
I20260928 22:48:28.690218 20995 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:28.690267 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:28.690392 20995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:28.691452 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "9b42bb2992db466d824a98c9e8ce9ded"
format_stamp: "Formatted at 2026-09-28 22:48:28 on dist-test-slave-pxr1"
server_key: "d268172b0b8aff38e8992a0203463cdc"
server_key_iv: "ebd20e6445e952d628ac920782ef930a"
server_key_version: "encryptionkey@0"
I20260928 22:48:28.691776 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b42bb2992db466d824a98c9e8ce9ded"
format_stamp: "Formatted at 2026-09-28 22:48:28 on dist-test-slave-pxr1"
server_key: "d268172b0b8aff38e8992a0203463cdc"
server_key_iv: "ebd20e6445e952d628ac920782ef930a"
server_key_version: "encryptionkey@0"
I20260928 22:48:28.693096 20995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:28.693853 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:28.694060 20995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:28.694146 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "9b42bb2992db466d824a98c9e8ce9ded"
format_stamp: "Formatted at 2026-09-28 22:48:28 on dist-test-slave-pxr1"
server_key: "d268172b0b8aff38e8992a0203463cdc"
server_key_iv: "ebd20e6445e952d628ac920782ef930a"
server_key_version: "encryptionkey@0"
I20260928 22:48:28.694208 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:28.704668 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:28.707027 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:28.707180 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:28.711508 20995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260928 22:48:28.711584 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260928 22:48:28.711931 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:28.712949 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:28.715224 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded: Bootstrap starting.
I20260928 22:48:28.715852 21063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:28.716109 21063 log.cc:824] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:28.716912 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded: No bootstrap required, opened a new log
I20260928 22:48:28.717875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 20995
I20260928 22:48:28.718010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:28.718184 392 external_mini_cluster.cc:1428] Setting key f8423d0121a0d512c2b30028296c16f6
I20260928 22:48:28.718918 21063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260928 22:48:28.719064 21063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:28.719086 21063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b42bb2992db466d824a98c9e8ce9ded, State: Initialized, Role: FOLLOWER
I20260928 22:48:28.719200 21063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260928 22:48:28.719280 21063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:28.719308 21063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:28.719357 21063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:28.719918 21063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260928 22:48:28.720006 21063 leader_election.cc:302] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b42bb2992db466d824a98c9e8ce9ded; no voters:
I20260928 22:48:28.720148 21063 leader_election.cc:288] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:28.720201 21068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:28.720381 21068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [term 1 LEADER]: Becoming Leader. State: Replica: 9b42bb2992db466d824a98c9e8ce9ded, State: Running, Role: LEADER
I20260928 22:48:28.720388 21063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:28.720497 21068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260928 22:48:28.721021 21070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b42bb2992db466d824a98c9e8ce9ded. Latest consensus state: current_term: 1 leader_uuid: "9b42bb2992db466d824a98c9e8ce9ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260928 22:48:28.721135 21070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:28.721009 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b42bb2992db466d824a98c9e8ce9ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b42bb2992db466d824a98c9e8ce9ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260928 22:48:28.721325 21077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:28.721994 21077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:28.722633 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:28.723501 21077 catalog_manager.cc:1347] Generated new cluster ID: fc32575d519641f7b14ce6448f72eaf3
I20260928 22:48:28.723546 21077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:28.725323 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:28.718718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40385 (local address 127.0.0.1:39121)
0928 22:48:28.718884 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:28.718893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:28.718925 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:48:28.719023 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:28.719029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:28.719075 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:28.719215 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:28.719464 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:28.722002 (+ 2538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:28.723188 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:28.723738 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:28.723838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:28.724544 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:28.724548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:28.724548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:28.724782 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:28.724817 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:28.724831 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:28.724885 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:28.724980 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:48:28.725122 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 22:48:28.726166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:28.728534 21077 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:28.729094 21077 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:28.731578 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:28.727955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53917 (local address 127.0.0.1:39121)
0928 22:48:28.728012 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:28.728014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:28.728085 (+ 71us) server_negotiation.cc:407] Connection header received
0928 22:48:28.728315 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:28.728316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:28.728326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:28.728381 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:28.728584 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:28.729346 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:28.730311 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:28.730600 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:28.730672 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:28.731328 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:28.731329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:28.731329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:28.731335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:28.731351 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:28.731356 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:28.731404 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:28.731456 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:48:28.731499 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:28.740942 21077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b42bb2992db466d824a98c9e8ce9ded: Generated new TSK 0
I20260928 22:48:28.741137 21077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:28.742763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 20995
2026-09-28T22:48:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:28 dist-test-slave-pxr1 krb5kdc[21094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:28 dist-test-slave-pxr1 krb5kdc[21094](info): set up 2 sockets
Sep 28 22:48:28 dist-test-slave-pxr1 krb5kdc[21094](info): commencing operation
krb5kdc: starting...
W20260928 22:48:29.692601 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:29 dist-test-slave-pxr1 krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:29.801720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:29.879817 21107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:29.880013 21107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:29.880035 21107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:29.881229 21107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:29.881274 21107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:29.881287 21107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:29.881305 21107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:29.881316 21107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:29.882777 21107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:29.883002 21107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:29.883206 21107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:29.887733 21115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:29.887739 21113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:29.887820 21107 server_base.cc:1034] running on GCE node
W20260928 22:48:29.887739 21112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:29.888098 21107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:29.888360 21107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:29.889518 21107 hybrid_clock.cc:648] HybridClock initialized: now 1790635709889502 us; error 27 us; skew 500 ppm
Sep 28 22:48:29 dist-test-slave-pxr1 krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:29.891939 21107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:29.892362 21107 webserver.cc:460] Webserver started at http://127.0.0.1:40229/ using document root <none> and password file <none>
I20260928 22:48:29.892557 21107 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:29.892604 21107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:29.892756 21107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:29.893826 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "8faafa7ea50949218bd0d34fc9ab6a29"
format_stamp: "Formatted at 2026-09-28 22:48:29 on dist-test-slave-pxr1"
server_key: "b97c2b3470bfc988e0e1b897f6098429"
server_key_iv: "b1c63bab362f601761c5e49f85f2bef0"
server_key_version: "encryptionkey@0"
I20260928 22:48:29.894162 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "8faafa7ea50949218bd0d34fc9ab6a29"
format_stamp: "Formatted at 2026-09-28 22:48:29 on dist-test-slave-pxr1"
server_key: "b97c2b3470bfc988e0e1b897f6098429"
server_key_iv: "b1c63bab362f601761c5e49f85f2bef0"
server_key_version: "encryptionkey@0"
I20260928 22:48:29.895408 21107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:48:29.896054 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:29.896184 21107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:29.896265 21107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "8faafa7ea50949218bd0d34fc9ab6a29"
format_stamp: "Formatted at 2026-09-28 22:48:29 on dist-test-slave-pxr1"
server_key: "b97c2b3470bfc988e0e1b897f6098429"
server_key_iv: "b1c63bab362f601761c5e49f85f2bef0"
server_key_version: "encryptionkey@0"
I20260928 22:48:29.896325 21107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:29.899497 21107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:29.901364 21107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:29.901489 21107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:29.905069 21107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39491
I20260928 22:48:29.905100 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39491 every 8 connection(s)
I20260928 22:48:29.905417 21107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:29.905656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21107
I20260928 22:48:29.905779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:29.905965 392 external_mini_cluster.cc:1428] Setting key 9356011e5a95e3a2cacb92bddc23ae03
I20260928 22:48:29.906155 21175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:29.908356 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:29.906466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39009 (local address 127.0.0.1:39491)
0928 22:48:29.906661 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:29.906670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:29.906703 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:48:29.907228 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:29.907237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:29.907273 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:29.907420 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:29.907967 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:29.907971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:29.907972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:29.907977 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:29.908018 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:29.908032 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:29.908109 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:29.908139 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 22:48:29.908177 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260928 22:48:29.909297 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:29.909461 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29: Bootstrap starting.
I20260928 22:48:29.910010 21175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:29.910305 21175 log.cc:824] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:29.911039 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29: No bootstrap required, opened a new log
I20260928 22:48:29.911537 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:29.910595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48933 (local address 127.0.0.1:39491)
0928 22:48:29.910632 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:29.910633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:29.910706 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:48:29.910763 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:29.910764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:29.910768 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:29.910837 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:29.911343 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:29.911345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:29.911345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:29.911349 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:29.911364 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:29.911368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:29.911438 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:29.911470 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:48:29.911482 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:29.913141 21175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260928 22:48:29.913290 21175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:29.913317 21175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8faafa7ea50949218bd0d34fc9ab6a29, State: Initialized, Role: FOLLOWER
I20260928 22:48:29.913427 21175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260928 22:48:29.913513 21175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:29.913537 21175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:29.913590 21175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:29.914134 21175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260928 22:48:29.914227 21175 leader_election.cc:302] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8faafa7ea50949218bd0d34fc9ab6a29; no voters:
I20260928 22:48:29.914403 21175 leader_election.cc:288] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:29.914495 21185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:29.914646 21175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:29.914660 21185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [term 1 LEADER]: Becoming Leader. State: Replica: 8faafa7ea50949218bd0d34fc9ab6a29, State: Running, Role: LEADER
I20260928 22:48:29.914816 21185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260928 22:48:29.915295 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260928 22:48:29.915266 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8faafa7ea50949218bd0d34fc9ab6a29. Latest consensus state: current_term: 1 leader_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faafa7ea50949218bd0d34fc9ab6a29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260928 22:48:29.915380 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:29.915387 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:29.915835 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:29.916303 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:29.917758 21191 catalog_manager.cc:1347] Generated new cluster ID: f78eb518e22a4438ab5a0049ebac7429
I20260928 22:48:29.917810 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:29.937026 21191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:29.937216 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:29.945052 21191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8faafa7ea50949218bd0d34fc9ab6a29: Generated new TSK 0
I20260928 22:48:29.945272 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:29.955163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21107
2026-09-28T22:48:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:30 dist-test-slave-pxr1 krb5kdc[21206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:30 dist-test-slave-pxr1 krb5kdc[21206](info): set up 2 sockets
Sep 28 22:48:30 dist-test-slave-pxr1 krb5kdc[21206](info): commencing operation
krb5kdc: starting...
W20260928 22:48:30.896522 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:30 dist-test-slave-pxr1 krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:31.008832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:31.087258 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:31.087473 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:31.087495 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:31.088665 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:31.088708 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:31.088766 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:31.088790 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:31.088814 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:31.090337 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:31.090569 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:31.090793 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:31.095364 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:31.095364 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:31.095422 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:31.095573 21219 server_base.cc:1034] running on GCE node
I20260928 22:48:31.095830 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:31.096077 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:31.097203 21219 hybrid_clock.cc:648] HybridClock initialized: now 1790635711097185 us; error 34 us; skew 500 ppm
Sep 28 22:48:31 dist-test-slave-pxr1 krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:31.099592 21219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:31.100016 21219 webserver.cc:460] Webserver started at http://127.0.0.1:42303/ using document root <none> and password file <none>
I20260928 22:48:31.100203 21219 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:31.100246 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:31.100359 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:31.101402 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "9ffa7082e7294fe995700061bf9bf047"
format_stamp: "Formatted at 2026-09-28 22:48:31 on dist-test-slave-pxr1"
server_key: "b15347053f053d442ef4c8e6c97f79e0"
server_key_iv: "62eb6159779739bddff832dac03c533c"
server_key_version: "encryptionkey@0"
I20260928 22:48:31.101712 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ffa7082e7294fe995700061bf9bf047"
format_stamp: "Formatted at 2026-09-28 22:48:31 on dist-test-slave-pxr1"
server_key: "b15347053f053d442ef4c8e6c97f79e0"
server_key_iv: "62eb6159779739bddff832dac03c533c"
server_key_version: "encryptionkey@0"
I20260928 22:48:31.102965 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:31.103657 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:31.103849 21219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:31.103940 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "9ffa7082e7294fe995700061bf9bf047"
format_stamp: "Formatted at 2026-09-28 22:48:31 on dist-test-slave-pxr1"
server_key: "b15347053f053d442ef4c8e6c97f79e0"
server_key_iv: "62eb6159779739bddff832dac03c533c"
server_key_version: "encryptionkey@0"
I20260928 22:48:31.103995 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:31.124689 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:31.126526 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:31.126641 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:31.130208 21219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44445
I20260928 22:48:31.130251 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44445 every 8 connection(s)
I20260928 22:48:31.130566 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:31.131392 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:31.133217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21219
I20260928 22:48:31.133353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:31.133553 392 external_mini_cluster.cc:1428] Setting key 9b796d2f152f176e04dee2cce35553ca
I20260928 22:48:31.133900 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: Bootstrap starting.
I20260928 22:48:31.134500 21287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:31.134752 21287 log.cc:824] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:31.135357 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: No bootstrap required, opened a new log
I20260928 22:48:31.137621 21287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } }
I20260928 22:48:31.137799 21287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:31.137840 21287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffa7082e7294fe995700061bf9bf047, State: Initialized, Role: FOLLOWER
I20260928 22:48:31.137966 21287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } }
I20260928 22:48:31.138036 21287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:31.138070 21287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:31.138113 21287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:31.138727 21287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } }
I20260928 22:48:31.138839 21287 leader_election.cc:302] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ffa7082e7294fe995700061bf9bf047; no voters:
I20260928 22:48:31.138989 21287 leader_election.cc:288] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:31.139026 21292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:31.139143 21292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [term 1 LEADER]: Becoming Leader. State: Replica: 9ffa7082e7294fe995700061bf9bf047, State: Running, Role: LEADER
I20260928 22:48:31.139222 21292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } }
I20260928 22:48:31.139253 21287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:31.139552 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffa7082e7294fe995700061bf9bf047. Latest consensus state: current_term: 1 leader_uuid: "9ffa7082e7294fe995700061bf9bf047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } } }
I20260928 22:48:31.139637 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:31.139565 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ffa7082e7294fe995700061bf9bf047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa7082e7294fe995700061bf9bf047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44445 } } }
I20260928 22:48:31.139783 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:48:31 dist-test-slave-pxr1 krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260928 22:48:31.141052 21308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:31.141273 21308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:31.141471 21298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:31.141633 21298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:31.143455 21298 catalog_manager.cc:1347] Generated new cluster ID: 0dbf4e014bc045df9d1158a35985c73d
I20260928 22:48:31.143503 21298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:31.144981 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:31.134051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:34085 (local address 127.0.0.1:44445)
0928 22:48:31.134236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:31.134244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:31.134270 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:48:31.134708 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:31.134715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:31.134763 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:31.134906 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:31.135536 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:31.136924 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:31.137639 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:31.138222 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:31.138343 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:31.140935 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:31.140939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:31.140940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:31.140953 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:31.142834 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:31.143260 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:31.143264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:31.143264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:31.143369 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:31.143640 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:31.143641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:31.143641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:31.143957 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:31.144114 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:31.144619 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 22:48:31.144755 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 22:48:31.145925 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:31.154467 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:31.147290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:55317 (local address 127.0.0.1:44445)
0928 22:48:31.147327 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:31.147329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:31.147536 (+ 207us) server_negotiation.cc:407] Connection header received
0928 22:48:31.147581 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:31.147582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:31.147592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:31.147644 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:31.148196 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:31.148992 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:31.149949 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:31.150283 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:31.150354 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:31.151670 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:31.151671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:31.151672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:31.151679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:31.153212 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:31.153619 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:31.153620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:31.153621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:31.153677 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:31.153882 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:31.153882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:31.153883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:31.154005 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:31.154118 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:31.154255 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:48:31.154316 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:31.154683 21298 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:31.155170 21298 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:31.159276 21298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ffa7082e7294fe995700061bf9bf047: Generated new TSK 0
I20260928 22:48:31.159453 21298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:31.165763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21219
W20260928 22:48:31.169544 21204 connection.cc:535] client connection to 127.0.0.1:44445 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44445): Connection reset by peer (error 104)
2026-09-28T22:48:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:31 dist-test-slave-pxr1 krb5kdc[21318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:31 dist-test-slave-pxr1 krb5kdc[21318](info): set up 2 sockets
Sep 28 22:48:31 dist-test-slave-pxr1 krb5kdc[21318](info): commencing operation
krb5kdc: starting...
W20260928 22:48:32.120534 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:32 dist-test-slave-pxr1 krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:32.234905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:32.315316 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:32.315526 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:32.315549 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:32.316685 21331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:32.316754 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:32.316794 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:32.316811 21331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:32.316833 21331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:32.318310 21331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:32.318495 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:32.318668 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:32.323249 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:32.323239 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:32.323239 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:32.323254 21331 server_base.cc:1034] running on GCE node
I20260928 22:48:32.323731 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:32.324025 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:32.325169 21331 hybrid_clock.cc:648] HybridClock initialized: now 1790635712325154 us; error 31 us; skew 500 ppm
Sep 28 22:48:32 dist-test-slave-pxr1 krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:32.327478 21331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:32.327889 21331 webserver.cc:460] Webserver started at http://127.0.0.1:46547/ using document root <none> and password file <none>
I20260928 22:48:32.328078 21331 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:32.328122 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:32.328243 21331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:32.329321 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "7af2863242374ac3a82783dc9ecdf80d"
format_stamp: "Formatted at 2026-09-28 22:48:32 on dist-test-slave-pxr1"
server_key: "9715a9acf7a00210a880bfe637171a12"
server_key_iv: "d446a9ed6890c0c9b7362861af7312d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:32.329679 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "7af2863242374ac3a82783dc9ecdf80d"
format_stamp: "Formatted at 2026-09-28 22:48:32 on dist-test-slave-pxr1"
server_key: "9715a9acf7a00210a880bfe637171a12"
server_key_iv: "d446a9ed6890c0c9b7362861af7312d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:32.331199 21331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:32.331804 21346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:32.331948 21331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:32.332029 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "7af2863242374ac3a82783dc9ecdf80d"
format_stamp: "Formatted at 2026-09-28 22:48:32 on dist-test-slave-pxr1"
server_key: "9715a9acf7a00210a880bfe637171a12"
server_key_iv: "d446a9ed6890c0c9b7362861af7312d0"
server_key_version: "encryptionkey@0"
I20260928 22:48:32.332088 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:32.343762 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:32.345562 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:32.345706 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:32.349432 21331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40037
I20260928 22:48:32.349475 21398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40037 every 8 connection(s)
I20260928 22:48:32.349804 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:32.350574 21399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:32.352741 21399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d: Bootstrap starting.
I20260928 22:48:32.353292 21399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:32.353530 21399 log.cc:824] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:32.354158 21399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d: No bootstrap required, opened a new log
I20260928 22:48:32.355662 21399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260928 22:48:32.355799 21399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:32.355831 21399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af2863242374ac3a82783dc9ecdf80d, State: Initialized, Role: FOLLOWER
I20260928 22:48:32.355971 21399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260928 22:48:32.356045 21399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:32.356078 21399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:32.356122 21399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:32.356608 21399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260928 22:48:32.356693 21399 leader_election.cc:302] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af2863242374ac3a82783dc9ecdf80d; no voters:
I20260928 22:48:32.356873 21399 leader_election.cc:288] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:32.356914 21402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:32.357080 21402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [term 1 LEADER]: Becoming Leader. State: Replica: 7af2863242374ac3a82783dc9ecdf80d, State: Running, Role: LEADER
I20260928 22:48:32.357085 21399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:32.357239 21402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } }
I20260928 22:48:32.357601 21404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af2863242374ac3a82783dc9ecdf80d. Latest consensus state: current_term: 1 leader_uuid: "7af2863242374ac3a82783dc9ecdf80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } } }
I20260928 22:48:32.357609 21403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af2863242374ac3a82783dc9ecdf80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af2863242374ac3a82783dc9ecdf80d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40037 } } }
I20260928 22:48:32.357751 21404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:32.357831 21403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:32.358103 21408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:32.358579 21408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:32.359506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21331
I20260928 22:48:32.359628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:32.359843 392 external_mini_cluster.cc:1428] Setting key bd3f8386dd8a283a82aa95cc1d3d3038
I20260928 22:48:32.360216 21408 catalog_manager.cc:1347] Generated new cluster ID: 85ade0585f274659a8b67b49b45952ab
I20260928 22:48:32.360270 21408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:32.364959 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:32.360417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:44977 (local address 127.0.0.1:40037)
0928 22:48:32.361039 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:32.361046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:32.361069 (+ 23us) server_negotiation.cc:407] Connection header received
0928 22:48:32.361155 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:32.361160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:32.361203 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:32.361316 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:32.361520 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:32.362416 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:32.363160 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:32.363589 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:32.363680 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:32.364312 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:32.364316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:32.364316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:32.364476 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:32.364506 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:32.364516 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:32.364583 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:32.364670 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:48:32.364812 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":57,"threads_started":1}
I20260928 22:48:32.365756 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:32.370887 21408 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:32.371564 21408 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:32.373108 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:32.368989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:33913 (local address 127.0.0.1:40037)
0928 22:48:32.369023 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:32.369025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:32.370495 (+ 1470us) server_negotiation.cc:407] Connection header received
0928 22:48:32.370559 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:32.370560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:32.370571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:32.370641 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:32.370766 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:32.371305 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:32.371999 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:32.372219 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:32.372277 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:32.372883 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:32.372885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:32.372885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:32.372890 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:32.372904 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:32.372908 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:32.372956 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:32.373021 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:48:32.373032 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:32.381160 21408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af2863242374ac3a82783dc9ecdf80d: Generated new TSK 0
I20260928 22:48:32.381358 21408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:32.384457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21331
W20260928 22:48:32.388142 21316 connection.cc:535] client connection to 127.0.0.1:40037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40037): Connection reset by peer (error 104)
2026-09-28T22:48:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:32 dist-test-slave-pxr1 krb5kdc[21430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:32 dist-test-slave-pxr1 krb5kdc[21430](info): set up 2 sockets
Sep 28 22:48:32 dist-test-slave-pxr1 krb5kdc[21430](info): commencing operation
krb5kdc: starting...
W20260928 22:48:33.324509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:33 dist-test-slave-pxr1 krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:33.434950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:33.514374 21443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:33.514590 21443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:33.514622 21443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:33.515780 21443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:33.515823 21443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:33.515841 21443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:33.515858 21443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:33.515874 21443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:33.517369 21443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:33.517616 21443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:33.517813 21443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:33.522313 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:33.522395 21443 server_base.cc:1034] running on GCE node
W20260928 22:48:33.522320 21448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:33.522320 21451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:33.522758 21443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:33.523033 21443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:33.524186 21443 hybrid_clock.cc:648] HybridClock initialized: now 1790635713524170 us; error 34 us; skew 500 ppm
Sep 28 22:48:33 dist-test-slave-pxr1 krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:33.526556 21443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:33.526958 21443 webserver.cc:460] Webserver started at http://127.0.0.1:45553/ using document root <none> and password file <none>
I20260928 22:48:33.527141 21443 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:33.527191 21443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:33.527281 21443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:33.528264 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "8db9fc8da36748b2b80b271b10651e96"
format_stamp: "Formatted at 2026-09-28 22:48:33 on dist-test-slave-pxr1"
server_key: "af6ed44346c4f4d8c4297ec8cb97523f"
server_key_iv: "1e1949b7492b6c095d2ae5f187640395"
server_key_version: "encryptionkey@0"
I20260928 22:48:33.528543 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "8db9fc8da36748b2b80b271b10651e96"
format_stamp: "Formatted at 2026-09-28 22:48:33 on dist-test-slave-pxr1"
server_key: "af6ed44346c4f4d8c4297ec8cb97523f"
server_key_iv: "1e1949b7492b6c095d2ae5f187640395"
server_key_version: "encryptionkey@0"
I20260928 22:48:33.529714 21443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:33.530323 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:33.530465 21443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:33.530552 21443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "8db9fc8da36748b2b80b271b10651e96"
format_stamp: "Formatted at 2026-09-28 22:48:33 on dist-test-slave-pxr1"
server_key: "af6ed44346c4f4d8c4297ec8cb97523f"
server_key_iv: "1e1949b7492b6c095d2ae5f187640395"
server_key_version: "encryptionkey@0"
I20260928 22:48:33.530615 21443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:33.533769 21443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:33.535774 21443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:33.535893 21443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:33.539541 21443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40333
I20260928 22:48:33.539570 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40333 every 8 connection(s)
I20260928 22:48:33.539865 21443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:33.540795 21511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:33.542977 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96: Bootstrap starting.
I20260928 22:48:33.543495 21511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:33.543730 21511 log.cc:824] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:33.544644 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96: No bootstrap required, opened a new log
I20260928 22:48:33.546258 21511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } }
I20260928 22:48:33.546399 21511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:33.546420 21511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db9fc8da36748b2b80b271b10651e96, State: Initialized, Role: FOLLOWER
I20260928 22:48:33.546545 21511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } }
I20260928 22:48:33.546617 21511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:33.546643 21511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:33.546679 21511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:33.547196 21511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } }
I20260928 22:48:33.547281 21511 leader_election.cc:302] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db9fc8da36748b2b80b271b10651e96; no voters:
I20260928 22:48:33.547420 21511 leader_election.cc:288] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:33.547506 21514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:33.547665 21514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [term 1 LEADER]: Becoming Leader. State: Replica: 8db9fc8da36748b2b80b271b10651e96, State: Running, Role: LEADER
I20260928 22:48:33.547673 21511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:33.547809 21514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } }
I20260928 22:48:33.548183 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db9fc8da36748b2b80b271b10651e96. Latest consensus state: current_term: 1 leader_uuid: "8db9fc8da36748b2b80b271b10651e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } } }
I20260928 22:48:33.548309 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db9fc8da36748b2b80b271b10651e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db9fc8da36748b2b80b271b10651e96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40333 } } }
I20260928 22:48:33.548412 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:33.548308 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:33.548743 21523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:33.549198 21523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:33.549431 21531 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:48:33.549496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21443
I20260928 22:48:33.549635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:33.549816 392 external_mini_cluster.cc:1428] Setting key 8544fe696ceedef2ee0354e2e1bd7815
I20260928 22:48:33.551013 21523 catalog_manager.cc:1347] Generated new cluster ID: 7980527678d340549a20b2e0bf77e521
I20260928 22:48:33.551110 21523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:33.553683 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:33.550300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:56469 (local address 127.0.0.1:40333)
0928 22:48:33.550574 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:33.550581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:33.550610 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:48:33.550727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:33.550734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:33.550775 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:33.550917 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:33.553255 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:33.553259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:33.553260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:33.553264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:33.553314 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:33.553330 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:33.553386 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:33.553457 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:48:33.553501 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":68,"threads_started":1}
I20260928 22:48:33.554593 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:33.557283 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:33.556131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:35841 (local address 127.0.0.1:40333)
0928 22:48:33.556198 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:33.556201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:33.556347 (+ 146us) server_negotiation.cc:407] Connection header received
0928 22:48:33.556397 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:33.556398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:33.556413 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:33.556464 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:33.557076 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:33.557077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:33.557077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:33.557081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:33.557101 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:33.557105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:33.557147 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:33.557212 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 22:48:33.557225 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:48:33.561933 21523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:33.562096 21523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:33.565302 21523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db9fc8da36748b2b80b271b10651e96: Generated new TSK 0
I20260928 22:48:33.565464 21523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:33.568840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21443
2026-09-28T22:48:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:33 dist-test-slave-pxr1 krb5kdc[21542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:33 dist-test-slave-pxr1 krb5kdc[21542](info): set up 2 sockets
Sep 28 22:48:33 dist-test-slave-pxr1 krb5kdc[21542](info): commencing operation
krb5kdc: starting...
W20260928 22:48:34.524583 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:34.635968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:34.715394 21555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:34.715615 21555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:34.715646 21555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:34.716907 21555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:34.716955 21555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:34.716975 21555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:34.716991 21555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:34.717008 21555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:34.718587 21555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:34.718843 21555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:34.719063 21555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:34.723724 21563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:34.723724 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:34.723730 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:34.723973 21555 server_base.cc:1034] running on GCE node
I20260928 22:48:34.724203 21555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:34.724483 21555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:34.725646 21555 hybrid_clock.cc:648] HybridClock initialized: now 1790635714725613 us; error 51 us; skew 500 ppm
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:34.734159 21555 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:34.734649 21555 webserver.cc:460] Webserver started at http://127.0.0.1:39729/ using document root <none> and password file <none>
I20260928 22:48:34.734884 21555 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:34.734941 21555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:34.735075 21555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:34.736157 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "cdc45209410246ac9eb2eca689ff135a"
format_stamp: "Formatted at 2026-09-28 22:48:34 on dist-test-slave-pxr1"
server_key: "e8ca394f898a5e173b76800f7a39fc5d"
server_key_iv: "b6ac972cbb6cd8377977aef8636f79ef"
server_key_version: "encryptionkey@0"
I20260928 22:48:34.736511 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdc45209410246ac9eb2eca689ff135a"
format_stamp: "Formatted at 2026-09-28 22:48:34 on dist-test-slave-pxr1"
server_key: "e8ca394f898a5e173b76800f7a39fc5d"
server_key_iv: "b6ac972cbb6cd8377977aef8636f79ef"
server_key_version: "encryptionkey@0"
I20260928 22:48:34.737895 21555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:48:34.738762 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:34.738956 21555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:34.739049 21555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "cdc45209410246ac9eb2eca689ff135a"
format_stamp: "Formatted at 2026-09-28 22:48:34 on dist-test-slave-pxr1"
server_key: "e8ca394f898a5e173b76800f7a39fc5d"
server_key_iv: "b6ac972cbb6cd8377977aef8636f79ef"
server_key_version: "encryptionkey@0"
I20260928 22:48:34.739122 21555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:34.757553 21555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:34.759426 21555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:34.759567 21555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:34.763353 21555 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34371
I20260928 22:48:34.763388 21622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34371 every 8 connection(s)
I20260928 22:48:34.763729 21555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:34.764524 21623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:34.766773 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a: Bootstrap starting.
I20260928 22:48:34.767313 21623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:34.767561 21623 log.cc:824] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:34.768200 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a: No bootstrap required, opened a new log
I20260928 22:48:34.769798 21623 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260928 22:48:34.769935 21623 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:34.769968 21623 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc45209410246ac9eb2eca689ff135a, State: Initialized, Role: FOLLOWER
I20260928 22:48:34.770107 21623 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260928 22:48:34.770164 21623 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:34.770195 21623 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:34.770239 21623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:34.770735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21555
I20260928 22:48:34.770862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:34.770855 21623 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260928 22:48:34.770951 21623 leader_election.cc:302] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc45209410246ac9eb2eca689ff135a; no voters:
I20260928 22:48:34.771040 392 external_mini_cluster.cc:1428] Setting key c2e01365a3a0743d115caa255013d677
I20260928 22:48:34.771116 21623 leader_election.cc:288] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:34.771152 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:34.771310 21626 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [term 1 LEADER]: Becoming Leader. State: Replica: cdc45209410246ac9eb2eca689ff135a, State: Running, Role: LEADER
I20260928 22:48:34.771365 21623 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:34.771431 21626 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260928 22:48:34.771808 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc45209410246ac9eb2eca689ff135a. Latest consensus state: current_term: 1 leader_uuid: "cdc45209410246ac9eb2eca689ff135a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260928 22:48:34.771835 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc45209410246ac9eb2eca689ff135a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc45209410246ac9eb2eca689ff135a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260928 22:48:34.771955 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:34.771963 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:34.772188 21637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:34.772697 21637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:34.777371 21637 catalog_manager.cc:1347] Generated new cluster ID: 07a9ed15d2be4e60addf3ac7585374ac
I20260928 22:48:34.777427 21637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:48:34.781468 21637 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:34.782121 21637 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:34.782418 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:34.771941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:54787 (local address 127.0.0.1:34371)
0928 22:48:34.772199 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:34.772207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:34.772243 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:48:34.772370 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:34.772378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:34.772447 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:34.772604 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:34.773419 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:34.774362 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:34.775433 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:34.775845 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:34.775920 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:34.778675 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:34.778679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:34.778680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:34.778704 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:34.780653 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:34.781141 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:34.781144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:34.781145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:34.781258 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:34.781501 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:34.781502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:34.781502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:34.781789 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:34.781961 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:34.782097 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:48:34.782214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":69,"threads_started":1}
I20260928 22:48:34.783231 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:34.787776 21637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc45209410246ac9eb2eca689ff135a: Generated new TSK 0
I20260928 22:48:34.787958 21637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:34.791208 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:34.784631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:48823 (local address 127.0.0.1:34371)
0928 22:48:34.784675 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:34.784677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:34.784810 (+ 133us) server_negotiation.cc:407] Connection header received
0928 22:48:34.784893 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:34.784894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:34.784903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:34.784926 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:34.785543 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:34.786313 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:34.787321 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:34.787583 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:34.787624 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:34.788929 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:34.788931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:34.788931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:34.788939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:34.790142 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:34.790487 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:34.790488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:34.790489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:34.790538 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:34.790726 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:34.790727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:34.790727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:34.790827 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:34.790918 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:34.791059 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 22:48:34.791119 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:48:34.793030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21555
W20260928 22:48:34.796953 21540 connection.cc:535] client connection to 127.0.0.1:34371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34371): Connection reset by peer (error 104)
2026-09-28T22:48:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21654](info): set up 2 sockets
Sep 28 22:48:34 dist-test-slave-pxr1 krb5kdc[21654](info): commencing operation
krb5kdc: starting...
W20260928 22:48:35.736629 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:35 dist-test-slave-pxr1 krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:35.845019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:35.924129 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:35.924337 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:35.924357 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:35.925580 21667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:35.925637 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:35.925657 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:35.925671 21667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:35.925686 21667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:35.927479 21667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:35.927717 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:35.927932 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:35.933305 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:35.933317 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:35.933563 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:35.933774 21667 server_base.cc:1034] running on GCE node
I20260928 22:48:35.934041 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:35.934352 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:35.935494 21667 hybrid_clock.cc:648] HybridClock initialized: now 1790635715935471 us; error 38 us; skew 500 ppm
Sep 28 22:48:35 dist-test-slave-pxr1 krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:35.937885 21667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:35.938313 21667 webserver.cc:460] Webserver started at http://127.0.0.1:46709/ using document root <none> and password file <none>
I20260928 22:48:35.938516 21667 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:35.938563 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:35.938690 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:35.939718 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "b6c94c5e14b44458b74310f4039ab8d8"
format_stamp: "Formatted at 2026-09-28 22:48:35 on dist-test-slave-pxr1"
server_key: "6d2db9e67eef6569831d50f513369c06"
server_key_iv: "c43c3a34fedebf552658f1c248d99616"
server_key_version: "encryptionkey@0"
I20260928 22:48:35.940044 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6c94c5e14b44458b74310f4039ab8d8"
format_stamp: "Formatted at 2026-09-28 22:48:35 on dist-test-slave-pxr1"
server_key: "6d2db9e67eef6569831d50f513369c06"
server_key_iv: "c43c3a34fedebf552658f1c248d99616"
server_key_version: "encryptionkey@0"
I20260928 22:48:35.941323 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:35.941960 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:35.942106 21667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:35.942186 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "b6c94c5e14b44458b74310f4039ab8d8"
format_stamp: "Formatted at 2026-09-28 22:48:35 on dist-test-slave-pxr1"
server_key: "6d2db9e67eef6569831d50f513369c06"
server_key_iv: "c43c3a34fedebf552658f1c248d99616"
server_key_version: "encryptionkey@0"
I20260928 22:48:35.942246 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:35.965466 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:35.967612 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:35.967751 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:35.971804 21667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46625
I20260928 22:48:35.971836 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46625 every 8 connection(s)
I20260928 22:48:35.972174 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:35.973148 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:35.975391 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8: Bootstrap starting.
I20260928 22:48:35.975946 21735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:35.976197 21735 log.cc:824] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:35.976857 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8: No bootstrap required, opened a new log
I20260928 22:48:35.978399 21735 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260928 22:48:35.978543 21735 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:35.978565 21735 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c94c5e14b44458b74310f4039ab8d8, State: Initialized, Role: FOLLOWER
I20260928 22:48:35.978689 21735 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260928 22:48:35.978750 21735 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:35.978773 21735 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:35.978816 21735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:35.979326 21735 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260928 22:48:35.979413 21735 leader_election.cc:302] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c94c5e14b44458b74310f4039ab8d8; no voters:
I20260928 22:48:35.979573 21735 leader_election.cc:288] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:35.979592 21738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:35.979652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21667
I20260928 22:48:35.979701 21738 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [term 1 LEADER]: Becoming Leader. State: Replica: b6c94c5e14b44458b74310f4039ab8d8, State: Running, Role: LEADER
I20260928 22:48:35.979769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:35.979771 21738 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260928 22:48:35.979830 21735 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:35.979943 392 external_mini_cluster.cc:1428] Setting key 470793cc54c54f43a9377adf391cb62c
I20260928 22:48:35.980079 21738 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c94c5e14b44458b74310f4039ab8d8. Latest consensus state: current_term: 1 leader_uuid: "b6c94c5e14b44458b74310f4039ab8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260928 22:48:35.980137 21738 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:35.980265 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c94c5e14b44458b74310f4039ab8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c94c5e14b44458b74310f4039ab8d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260928 22:48:35.980345 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:35.981031 21742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:35.981554 21742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:35.983100 21742 catalog_manager.cc:1347] Generated new cluster ID: 46df6f07403847cfb50f1d05a1660845
I20260928 22:48:35.983146 21742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:35.985951 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:35.980407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:60297 (local address 127.0.0.1:46625)
0928 22:48:35.980686 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:35.980696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:35.980773 (+ 77us) server_negotiation.cc:407] Connection header received
0928 22:48:35.980967 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:35.980975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:35.981040 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:35.981284 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:35.981508 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:35.982651 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:35.983767 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:35.984382 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:35.984482 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:35.985287 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:35.985291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:35.985292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:35.985486 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:35.985523 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:35.985537 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:35.985618 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:35.985715 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 22:48:35.985744 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":64,"threads_started":1}
I20260928 22:48:35.986836 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:35.993414 21742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:35.994212 21742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:35.995843 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:35.989024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:41139 (local address 127.0.0.1:46625)
0928 22:48:35.989076 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:35.989078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:35.993128 (+ 4050us) server_negotiation.cc:407] Connection header received
0928 22:48:35.993185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:35.993186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:35.993198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:35.993263 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:35.993532 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:35.994004 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:35.994692 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:35.994913 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:35.994967 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:35.995609 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:35.995611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:35.995611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:35.995616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:35.995632 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:35.995638 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:35.995681 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:35.995751 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 22:48:35.995765 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:48:36.001343 21742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c94c5e14b44458b74310f4039ab8d8: Generated new TSK 0
I20260928 22:48:36.001528 21742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:36.004676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21667
W20260928 22:48:36.008378 21652 connection.cc:535] client connection to 127.0.0.1:46625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46625): Connection reset by peer (error 104)
2026-09-28T22:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:36 dist-test-slave-pxr1 krb5kdc[21766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:36 dist-test-slave-pxr1 krb5kdc[21766](info): set up 2 sockets
Sep 28 22:48:36 dist-test-slave-pxr1 krb5kdc[21766](info): commencing operation
krb5kdc: starting...
W20260928 22:48:36.944516 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:37 dist-test-slave-pxr1 krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:48:37.055091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:37.134217 21779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:37.134462 21779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:37.134496 21779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:37.135685 21779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:37.135725 21779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:37.135736 21779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:37.135748 21779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:37.135758 21779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:37.137252 21779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:37.137477 21779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:37.137667 21779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:37.142267 21787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:37.142315 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:37.142259 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:37.142339 21779 server_base.cc:1034] running on GCE node
I20260928 22:48:37.142737 21779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:37.143046 21779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:37.144227 21779 hybrid_clock.cc:648] HybridClock initialized: now 1790635717144204 us; error 28 us; skew 500 ppm
Sep 28 22:48:37 dist-test-slave-pxr1 krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:37.147186 21779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:48:37.147652 21779 webserver.cc:460] Webserver started at http://127.0.0.1:42443/ using document root <none> and password file <none>
I20260928 22:48:37.147879 21779 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:37.147929 21779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:37.148062 21779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:37.149250 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "58363914e304484aa33a9efcaa5803b8"
format_stamp: "Formatted at 2026-09-28 22:48:37 on dist-test-slave-pxr1"
server_key: "37c2e56bfe4beb892f1bea4a1fb6bd19"
server_key_iv: "297d26bd72a67c17aecdea377f4b6419"
server_key_version: "encryptionkey@0"
I20260928 22:48:37.149597 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "58363914e304484aa33a9efcaa5803b8"
format_stamp: "Formatted at 2026-09-28 22:48:37 on dist-test-slave-pxr1"
server_key: "37c2e56bfe4beb892f1bea4a1fb6bd19"
server_key_iv: "297d26bd72a67c17aecdea377f4b6419"
server_key_version: "encryptionkey@0"
I20260928 22:48:37.150966 21779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:37.151815 21794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:37.151984 21779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:37.152079 21779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "58363914e304484aa33a9efcaa5803b8"
format_stamp: "Formatted at 2026-09-28 22:48:37 on dist-test-slave-pxr1"
server_key: "37c2e56bfe4beb892f1bea4a1fb6bd19"
server_key_iv: "297d26bd72a67c17aecdea377f4b6419"
server_key_version: "encryptionkey@0"
I20260928 22:48:37.152154 21779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:37.155508 21779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:37.157684 21779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:37.157827 21779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:37.161583 21779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37427
I20260928 22:48:37.161648 21846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37427 every 8 connection(s)
I20260928 22:48:37.161959 21779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:37.162923 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:37.165184 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8: Bootstrap starting.
I20260928 22:48:37.165701 21847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:37.165922 21847 log.cc:824] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:37.166563 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8: No bootstrap required, opened a new log
I20260928 22:48:37.168081 21847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260928 22:48:37.168221 21847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:37.168251 21847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58363914e304484aa33a9efcaa5803b8, State: Initialized, Role: FOLLOWER
I20260928 22:48:37.168371 21847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260928 22:48:37.168442 21847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:37.168469 21847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:37.168501 21847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:37.169036 21847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260928 22:48:37.169121 21847 leader_election.cc:302] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58363914e304484aa33a9efcaa5803b8; no voters:
I20260928 22:48:37.169241 21847 leader_election.cc:288] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:37.169313 21850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:37.169479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21779
I20260928 22:48:37.169489 21847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:37.169498 21850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [term 1 LEADER]: Becoming Leader. State: Replica: 58363914e304484aa33a9efcaa5803b8, State: Running, Role: LEADER
I20260928 22:48:37.169608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:37.169631 21850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260928 22:48:37.169790 392 external_mini_cluster.cc:1428] Setting key 1de8cf41d461c1a30531c060359c9733
I20260928 22:48:37.169924 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58363914e304484aa33a9efcaa5803b8. Latest consensus state: current_term: 1 leader_uuid: "58363914e304484aa33a9efcaa5803b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260928 22:48:37.169984 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:37.169978 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58363914e304484aa33a9efcaa5803b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58363914e304484aa33a9efcaa5803b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260928 22:48:37.170058 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:37.170176 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:37.171033 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:37.172426 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:37.170486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:45475 (local address 127.0.0.1:37427)
0928 22:48:37.170644 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:37.170653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:37.170699 (+ 46us) server_negotiation.cc:407] Connection header received
0928 22:48:37.170901 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:37.170909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:37.170942 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:37.171444 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:37.171925 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:37.171928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:37.171929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:37.171932 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:37.171965 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:37.171980 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:37.172204 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:37.172238 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 22:48:37.172281 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260928 22:48:37.173297 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:37.175544 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:37.174576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:40133 (local address 127.0.0.1:37427)
0928 22:48:37.174616 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:37.174619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:37.174717 (+ 98us) server_negotiation.cc:407] Connection header received
0928 22:48:37.174776 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:37.174777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:37.174781 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:37.174833 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:37.175340 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:37.175341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:37.175341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:37.175346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:37.175360 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:37.175365 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:37.175410 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:37.175458 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 22:48:37.175472 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:37.177763 21854 catalog_manager.cc:1347] Generated new cluster ID: e0bf72c5ed994dc88e3dd4b559eec1ee
I20260928 22:48:37.177821 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:37.190265 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:37.190518 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:37.202473 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58363914e304484aa33a9efcaa5803b8: Generated new TSK 0
I20260928 22:48:37.202695 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:37.207753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21779
2026-09-28T22:48:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:37 dist-test-slave-pxr1 krb5kdc[21878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:37 dist-test-slave-pxr1 krb5kdc[21878](info): set up 2 sockets
Sep 28 22:48:37 dist-test-slave-pxr1 krb5kdc[21878](info): commencing operation
krb5kdc: starting...
W20260928 22:48:38.144652 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:38.253105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:38.330956 21891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:38.331180 21891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:38.331210 21891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:38.332408 21891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:38.332451 21891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:38.332469 21891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:38.332486 21891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:38.332500 21891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:38.334040 21891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:38.334286 21891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:38.334486 21891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:38.339020 21897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:38.339030 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:38.339218 21891 server_base.cc:1034] running on GCE node
W20260928 22:48:38.339039 21896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:38.339537 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:38.339807 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:38.340956 21891 hybrid_clock.cc:648] HybridClock initialized: now 1790635718340938 us; error 29 us; skew 500 ppm
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:38.343369 21891 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:38.343786 21891 webserver.cc:460] Webserver started at http://127.0.98.62:43273/ using document root <none> and password file <none>
I20260928 22:48:38.343981 21891 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:38.344024 21891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:38.344146 21891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:38.345225 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "a81acbf645a84c498283de4860d7af56"
format_stamp: "Formatted at 2026-09-28 22:48:38 on dist-test-slave-pxr1"
server_key: "628faa30099c41f90843f4ebf9416088"
server_key_iv: "247a4958f12e5831bd1972c861ebdae8"
server_key_version: "encryptionkey@0"
I20260928 22:48:38.345556 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "a81acbf645a84c498283de4860d7af56"
format_stamp: "Formatted at 2026-09-28 22:48:38 on dist-test-slave-pxr1"
server_key: "628faa30099c41f90843f4ebf9416088"
server_key_iv: "247a4958f12e5831bd1972c861ebdae8"
server_key_version: "encryptionkey@0"
I20260928 22:48:38.346830 21891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:38.347504 21906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:38.347656 21891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:38.347738 21891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "a81acbf645a84c498283de4860d7af56"
format_stamp: "Formatted at 2026-09-28 22:48:38 on dist-test-slave-pxr1"
server_key: "628faa30099c41f90843f4ebf9416088"
server_key_iv: "247a4958f12e5831bd1972c861ebdae8"
server_key_version: "encryptionkey@0"
I20260928 22:48:38.347800 21891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:38.358242 21891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:38.360045 21891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:38.360179 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:38.363863 21891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39941
I20260928 22:48:38.363903 21958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39941 every 8 connection(s)
I20260928 22:48:38.364225 21891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:38.365085 21959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:38.367223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 21891
I20260928 22:48:38.367286 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56: Bootstrap starting.
I20260928 22:48:38.367338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:38.367493 392 external_mini_cluster.cc:1428] Setting key 48a5801a23b66bd32269dec1d36b4aa2
I20260928 22:48:38.367895 21959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:38.368218 21959 log.cc:824] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:38.368904 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56: No bootstrap required, opened a new log
I20260928 22:48:38.371008 21959 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } }
I20260928 22:48:38.371178 21959 raft_consensus.cc:374] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:38.371209 21959 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81acbf645a84c498283de4860d7af56, State: Initialized, Role: FOLLOWER
I20260928 22:48:38.371335 21959 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } }
I20260928 22:48:38.371400 21959 raft_consensus.cc:388] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:38.371435 21959 raft_consensus.cc:482] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:38.371476 21959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:38.371989 21959 raft_consensus.cc:504] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } }
I20260928 22:48:38.372080 21959 leader_election.cc:302] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a81acbf645a84c498283de4860d7af56; no voters:
I20260928 22:48:38.372236 21959 leader_election.cc:288] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:38.372282 21964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:38.372429 21964 raft_consensus.cc:686] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [term 1 LEADER]: Becoming Leader. State: Replica: a81acbf645a84c498283de4860d7af56, State: Running, Role: LEADER
I20260928 22:48:38.372557 21964 consensus_queue.cc:237] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } }
I20260928 22:48:38.372825 21959 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:38.372965 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a81acbf645a84c498283de4860d7af56. Latest consensus state: current_term: 1 leader_uuid: "a81acbf645a84c498283de4860d7af56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } } }
I20260928 22:48:38.373039 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:38.373513 21968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:38.374004 21968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:38.374874 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a81acbf645a84c498283de4860d7af56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81acbf645a84c498283de4860d7af56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39941 } } }
I20260928 22:48:38.374989 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:38.377249 21968 catalog_manager.cc:1347] Generated new cluster ID: 1de1e4a6725d46a299264175e646e12e
I20260928 22:48:38.377298 21968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:38.378463 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:38.367983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49163 (local address 127.0.98.62:39941)
0928 22:48:38.368149 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:38.368155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:38.368180 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:48:38.368752 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:38.368762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:38.368828 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:38.368940 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:38.369694 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:38.370579 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:38.371600 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:38.372014 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:38.372118 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:38.375001 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:38.375005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:38.375005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:38.375017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:38.376748 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:38.377200 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:38.377203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:38.377203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:38.377316 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:38.377570 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:38.377571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:38.377572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:38.377864 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:38.378019 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:38.378147 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 22:48:38.378241 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 22:48:38.379400 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:38.387570 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:38.380777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57463 (local address 127.0.98.62:39941)
0928 22:48:38.380808 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:38.380810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:38.380895 (+ 85us) server_negotiation.cc:407] Connection header received
0928 22:48:38.380955 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:38.380956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:38.380964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:38.381011 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:38.381604 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:38.382308 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:38.383272 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:38.383581 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:38.383632 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:38.384987 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:38.384988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:38.384988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:38.384995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:38.386436 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:38.386830 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:38.386831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:38.386831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:38.386887 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:38.387090 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:38.387091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:38.387091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:38.387204 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:38.387291 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:38.387423 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 22:48:38.387483 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:38.388230 21968 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:38.388686 21968 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:38.397601 21968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a81acbf645a84c498283de4860d7af56: Generated new TSK 0
I20260928 22:48:38.397783 21968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:38.404634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 21891
W20260928 22:48:38.408335 21876 connection.cc:535] client connection to 127.0.98.62:39941 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39941): Connection reset by peer (error 104)
2026-09-28T22:48:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21990](info): set up 2 sockets
Sep 28 22:48:38 dist-test-slave-pxr1 krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20260928 22:48:39.348613 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:39 dist-test-slave-pxr1 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:39.458959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:39.537895 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:39.538158 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:39.538187 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:39.539384 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:39.539427 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:39.539440 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:39.539451 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:39.539466 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:39.541038 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:39.541249 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:39.541425 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:39.546234 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:39.546245 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:39.546270 22003 server_base.cc:1034] running on GCE node
W20260928 22:48:39.546276 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:39.546623 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:39.546933 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:39.548072 22003 hybrid_clock.cc:648] HybridClock initialized: now 1790635719548045 us; error 48 us; skew 500 ppm
Sep 28 22:48:39 dist-test-slave-pxr1 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:39.550464 22003 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:39.550910 22003 webserver.cc:460] Webserver started at http://127.0.98.62:37235/ using document root <none> and password file <none>
I20260928 22:48:39.551131 22003 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:39.551177 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:39.551291 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:39.552327 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f"
format_stamp: "Formatted at 2026-09-28 22:48:39 on dist-test-slave-pxr1"
server_key: "02d130b6df197286313c2031bf17b4f5"
server_key_iv: "8e6b55eb1de92d54e139e4403852e55b"
server_key_version: "encryptionkey@0"
I20260928 22:48:39.552646 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f"
format_stamp: "Formatted at 2026-09-28 22:48:39 on dist-test-slave-pxr1"
server_key: "02d130b6df197286313c2031bf17b4f5"
server_key_iv: "8e6b55eb1de92d54e139e4403852e55b"
server_key_version: "encryptionkey@0"
I20260928 22:48:39.553936 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:39.554661 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:39.554811 22003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:39.554886 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f"
format_stamp: "Formatted at 2026-09-28 22:48:39 on dist-test-slave-pxr1"
server_key: "02d130b6df197286313c2031bf17b4f5"
server_key_iv: "8e6b55eb1de92d54e139e4403852e55b"
server_key_version: "encryptionkey@0"
I20260928 22:48:39.554951 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:39.568125 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:39.570081 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:39.570215 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:39.574003 22003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40697
I20260928 22:48:39.574038 22070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40697 every 8 connection(s)
I20260928 22:48:39.574365 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:39.575176 22071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:39.577450 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f: Bootstrap starting.
I20260928 22:48:39.577975 22071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:39.578191 22071 log.cc:824] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:39.578801 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f: No bootstrap required, opened a new log
I20260928 22:48:39.580297 22071 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } }
I20260928 22:48:39.580435 22071 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:39.580457 22071 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe460b4c8b9b4a60a2b1cfd890cd6e1f, State: Initialized, Role: FOLLOWER
I20260928 22:48:39.580565 22071 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } }
I20260928 22:48:39.580616 22071 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:39.580636 22071 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:39.580660 22071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:39.581264 22071 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } }
I20260928 22:48:39.581351 22071 leader_election.cc:302] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe460b4c8b9b4a60a2b1cfd890cd6e1f; no voters:
I20260928 22:48:39.581475 22071 leader_election.cc:288] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:39.581522 22074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:39.581712 22074 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [term 1 LEADER]: Becoming Leader. State: Replica: fe460b4c8b9b4a60a2b1cfd890cd6e1f, State: Running, Role: LEADER
I20260928 22:48:39.581722 22071 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:39.581809 22074 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } }
I20260928 22:48:39.582350 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } } }
I20260928 22:48:39.582433 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:39.582598 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe460b4c8b9b4a60a2b1cfd890cd6e1f. Latest consensus state: current_term: 1 leader_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe460b4c8b9b4a60a2b1cfd890cd6e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40697 } } }
I20260928 22:48:39.582674 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:39.582777 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:39.583283 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:39.583294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22003
I20260928 22:48:39.583396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:39.583588 392 external_mini_cluster.cc:1428] Setting key 28fb1a9cf53358ac1b160a1b953d9edf
I20260928 22:48:39.585050 22083 catalog_manager.cc:1347] Generated new cluster ID: ae945cfa60b4468a8b1fa0c3910a422e
I20260928 22:48:39.585099 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:39.590152 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:39.584084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53633 (local address 127.0.98.62:40697)
0928 22:48:39.584329 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:39.584337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:39.584367 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:48:39.584479 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:39.584486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:39.584540 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:39.584672 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:39.584938 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:39.586350 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:39.587521 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:39.588159 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:39.588641 (+ 482us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:39.589079 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:39.589083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:39.589083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:39.589313 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:39.589356 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:39.589372 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:39.589673 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:39.589717 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:48:39.589748 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260928 22:48:39.591185 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:39.596037 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:39.592581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45603 (local address 127.0.98.62:40697)
0928 22:48:39.592625 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:39.592626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:39.592745 (+ 119us) server_negotiation.cc:407] Connection header received
0928 22:48:39.592850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:39.592851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:39.592861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:39.592905 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:39.593055 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:39.593783 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:39.594783 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:39.595093 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:39.595148 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:39.595806 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:39.595807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:39.595808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:39.595812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:39.595827 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:39.595832 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:39.595879 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:39.595946 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:48:39.595958 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:48:39.624015 22083 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:39.624541 22083 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:39.629946 22083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe460b4c8b9b4a60a2b1cfd890cd6e1f: Generated new TSK 0
I20260928 22:48:39.630138 22083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:39.633495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22003
W20260928 22:48:39.637668 21988 connection.cc:535] client connection to 127.0.98.62:40697 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40697): Connection reset by peer (error 104)
2026-09-28T22:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:39 dist-test-slave-pxr1 krb5kdc[22102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:39 dist-test-slave-pxr1 krb5kdc[22102](info): set up 2 sockets
Sep 28 22:48:39 dist-test-slave-pxr1 krb5kdc[22102](info): commencing operation
krb5kdc: starting...
W20260928 22:48:40.588531 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:40 dist-test-slave-pxr1 krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:40.698993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:40.780275 22115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:40.780504 22115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:40.780527 22115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:40.781726 22115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:40.781775 22115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:40.781797 22115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:40.781809 22115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:40.781821 22115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:40.783336 22115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:40.783550 22115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:40.783743 22115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:40.788430 22123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:40.788421 22121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:40.788688 22115 server_base.cc:1034] running on GCE node
W20260928 22:48:40.788420 22120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:40.789013 22115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:40.789311 22115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:40.790467 22115 hybrid_clock.cc:648] HybridClock initialized: now 1790635720790455 us; error 32 us; skew 500 ppm
Sep 28 22:48:40 dist-test-slave-pxr1 krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:40.801733 22115 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:40.802192 22115 webserver.cc:460] Webserver started at http://127.0.98.62:38809/ using document root <none> and password file <none>
I20260928 22:48:40.802415 22115 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:40.802465 22115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:40.802593 22115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:40.803704 22115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "c7f1b1ff9df54adfa868fb24543d46e2"
format_stamp: "Formatted at 2026-09-28 22:48:40 on dist-test-slave-pxr1"
server_key: "0386e3f84032612c5b3115324a56a393"
server_key_iv: "89ebbc0baa373659fee164c337d20578"
server_key_version: "encryptionkey@0"
I20260928 22:48:40.804031 22115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7f1b1ff9df54adfa868fb24543d46e2"
format_stamp: "Formatted at 2026-09-28 22:48:40 on dist-test-slave-pxr1"
server_key: "0386e3f84032612c5b3115324a56a393"
server_key_iv: "89ebbc0baa373659fee164c337d20578"
server_key_version: "encryptionkey@0"
I20260928 22:48:40.805330 22115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:48:40.805995 22130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:40.806144 22115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:40.806246 22115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "c7f1b1ff9df54adfa868fb24543d46e2"
format_stamp: "Formatted at 2026-09-28 22:48:40 on dist-test-slave-pxr1"
server_key: "0386e3f84032612c5b3115324a56a393"
server_key_iv: "89ebbc0baa373659fee164c337d20578"
server_key_version: "encryptionkey@0"
I20260928 22:48:40.806304 22115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:40.809497 22115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:40.811287 22115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:40.811408 22115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:40.815054 22115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34095
I20260928 22:48:40.815084 22182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34095 every 8 connection(s)
I20260928 22:48:40.815390 22115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:40.816294 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:40.818483 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2: Bootstrap starting.
I20260928 22:48:40.818987 22183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:40.819221 22183 log.cc:824] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:40.819808 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2: No bootstrap required, opened a new log
I20260928 22:48:40.821439 22183 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 22:48:40.821590 22183 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:40.821612 22183 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f1b1ff9df54adfa868fb24543d46e2, State: Initialized, Role: FOLLOWER
I20260928 22:48:40.821724 22183 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 22:48:40.821797 22183 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:40.821821 22183 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:40.821847 22183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:40.822368 22183 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 22:48:40.822451 22183 leader_election.cc:302] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f1b1ff9df54adfa868fb24543d46e2; no voters:
I20260928 22:48:40.822597 22183 leader_election.cc:288] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:40.822667 22186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:40.822825 22183 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:40.823019 22186 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [term 1 LEADER]: Becoming Leader. State: Replica: c7f1b1ff9df54adfa868fb24543d46e2, State: Running, Role: LEADER
I20260928 22:48:40.823119 22186 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260928 22:48:40.823462 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f1b1ff9df54adfa868fb24543d46e2. Latest consensus state: current_term: 1 leader_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260928 22:48:40.823563 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:40.823448 22187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f1b1ff9df54adfa868fb24543d46e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260928 22:48:40.823622 22187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:40.823742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22115
I20260928 22:48:40.823845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:40.824110 392 external_mini_cluster.cc:1428] Setting key 29acc9d26a184b06711b3f18607c89b9
I20260928 22:48:40.824523 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:40.825201 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:40.826669 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:40.824857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58797 (local address 127.0.98.62:34095)
0928 22:48:40.825016 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:40.825025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:40.825060 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:48:40.825188 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:40.825195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:40.825229 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:40.825674 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:40.826159 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:40.826162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:40.826162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:40.826165 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:40.826199 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:40.826213 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:40.826421 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:40.826456 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 22:48:40.826497 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260928 22:48:40.826984 22196 catalog_manager.cc:1347] Generated new cluster ID: aa9741991b9a4089955694f6fd557ad9
I20260928 22:48:40.827049 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:40.827508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:40.829821 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:40.828802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53315 (local address 127.0.98.62:34095)
0928 22:48:40.828837 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:40.828839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:40.828969 (+ 130us) server_negotiation.cc:407] Connection header received
0928 22:48:40.829025 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:40.829026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:40.829032 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:40.829076 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:40.829593 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:40.829594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:40.829594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:40.829598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:40.829613 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:40.829617 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:40.829664 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:40.829716 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:48:40.829730 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:40.834105 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:40.834277 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:40.844085 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f1b1ff9df54adfa868fb24543d46e2: Generated new TSK 0
I20260928 22:48:40.844297 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:40.845935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22115
2026-09-28T22:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:40 dist-test-slave-pxr1 krb5kdc[22214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:40 dist-test-slave-pxr1 krb5kdc[22214](info): set up 2 sockets
Sep 28 22:48:40 dist-test-slave-pxr1 krb5kdc[22214](info): commencing operation
krb5kdc: starting...
W20260928 22:48:41.792492 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:41 dist-test-slave-pxr1 krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:41.901544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:41.981909 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:41.982115 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:41.982146 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:41.983297 22227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:41.983340 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:41.983357 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:41.983374 22227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:41.983390 22227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:41.984900 22227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:41.985139 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:41.985352 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:41.989915 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:41.989964 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:41.989924 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:41.990170 22227 server_base.cc:1034] running on GCE node
I20260928 22:48:41.990391 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:41.990670 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:41.991813 22227 hybrid_clock.cc:648] HybridClock initialized: now 1790635721991800 us; error 29 us; skew 500 ppm
Sep 28 22:48:41 dist-test-slave-pxr1 krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:41.994154 22227 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:41.994570 22227 webserver.cc:460] Webserver started at http://127.0.98.62:40321/ using document root <none> and password file <none>
I20260928 22:48:41.994742 22227 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:41.994779 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:41.994869 22227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:41.995848 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "c0d8243598de4841b7c6fa810d46b8bb"
format_stamp: "Formatted at 2026-09-28 22:48:41 on dist-test-slave-pxr1"
server_key: "7a28d0a61239344b0a4335997c4ad64f"
server_key_iv: "3ff675a351a42ae47bb76135636d701c"
server_key_version: "encryptionkey@0"
I20260928 22:48:41.996129 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0d8243598de4841b7c6fa810d46b8bb"
format_stamp: "Formatted at 2026-09-28 22:48:41 on dist-test-slave-pxr1"
server_key: "7a28d0a61239344b0a4335997c4ad64f"
server_key_iv: "3ff675a351a42ae47bb76135636d701c"
server_key_version: "encryptionkey@0"
I20260928 22:48:41.997331 22227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:48:41.998000 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:41.998179 22227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:41.998277 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "c0d8243598de4841b7c6fa810d46b8bb"
format_stamp: "Formatted at 2026-09-28 22:48:41 on dist-test-slave-pxr1"
server_key: "7a28d0a61239344b0a4335997c4ad64f"
server_key_iv: "3ff675a351a42ae47bb76135636d701c"
server_key_version: "encryptionkey@0"
I20260928 22:48:41.998349 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:42.020193 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:42.022126 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:42.022260 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:42.025920 22227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40777
I20260928 22:48:42.025970 22294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40777 every 8 connection(s)
I20260928 22:48:42.026283 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:42.027112 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:42.029410 22295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb: Bootstrap starting.
I20260928 22:48:42.029963 22295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:42.030217 22295 log.cc:824] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:42.030819 22295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb: No bootstrap required, opened a new log
I20260928 22:48:42.032353 22295 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260928 22:48:42.032491 22295 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:42.032524 22295 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d8243598de4841b7c6fa810d46b8bb, State: Initialized, Role: FOLLOWER
I20260928 22:48:42.032658 22295 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260928 22:48:42.032771 22295 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:42.032809 22295 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:42.032856 22295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:42.033367 22295 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260928 22:48:42.033458 22295 leader_election.cc:302] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d8243598de4841b7c6fa810d46b8bb; no voters:
I20260928 22:48:42.033622 22295 leader_election.cc:288] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:42.033668 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:42.033811 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [term 1 LEADER]: Becoming Leader. State: Replica: c0d8243598de4841b7c6fa810d46b8bb, State: Running, Role: LEADER
I20260928 22:48:42.033847 22295 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:42.033897 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260928 22:48:42.034269 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d8243598de4841b7c6fa810d46b8bb. Latest consensus state: current_term: 1 leader_uuid: "c0d8243598de4841b7c6fa810d46b8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } } }
I20260928 22:48:42.034349 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:42.034466 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d8243598de4841b7c6fa810d46b8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8243598de4841b7c6fa810d46b8bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } } }
I20260928 22:48:42.034564 22307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:42.034564 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:42.035064 22307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:42.035822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22227
I20260928 22:48:42.035938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:42.036145 392 external_mini_cluster.cc:1428] Setting key 5002fa8c38131e6120691fb35660fc65
I20260928 22:48:42.037185 22307 catalog_manager.cc:1347] Generated new cluster ID: eb7bfb6f18d3462088e3e39587cb08d6
I20260928 22:48:42.037236 22307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:42 dist-test-slave-pxr1 krb5kdc[22214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:42.045610 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:42.036755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43471 (local address 127.0.98.62:40777)
0928 22:48:42.036889 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:42.036898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:42.036925 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:48:42.037015 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:42.037020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:42.037070 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:42.037228 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:42.037824 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:42.038726 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:42.039442 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:42.039832 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:42.039924 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:42.042396 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:42.042399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:42.042399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:42.042409 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:42.043892 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:42.044255 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:42.044257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:42.044258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:42.044347 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:42.044523 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:42.044523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:42.044523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:42.044782 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:42.045310 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:42.045367 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:48:42.045469 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260928 22:48:42.046530 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:42.053177 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:42.047753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33867 (local address 127.0.98.62:40777)
0928 22:48:42.047788 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:42.047790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:42.047894 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:48:42.047946 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:42.047947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:42.047956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:42.047995 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:42.048458 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:42.049013 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:42.049660 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:42.049862 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:42.049913 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:42.050991 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:42.050992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:42.050992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:42.050999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:42.052192 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:42.052475 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:42.052476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:42.052476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:42.052526 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:42.052682 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:42.052683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:42.052684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:42.052823 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:42.052919 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:42.053009 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:48:42.053090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:42.055276 22307 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:42.055845 22307 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:42.064294 22307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d8243598de4841b7c6fa810d46b8bb: Generated new TSK 0
I20260928 22:48:42.064489 22307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:42.073010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22227
2026-09-28T22:48:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:42 dist-test-slave-pxr1 krb5kdc[22326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:42 dist-test-slave-pxr1 krb5kdc[22326](info): set up 2 sockets
Sep 28 22:48:42 dist-test-slave-pxr1 krb5kdc[22326](info): commencing operation
krb5kdc: starting...
W20260928 22:48:43.021180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:43 dist-test-slave-pxr1 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:43.131683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:43.212121 22339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:43.212328 22339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:43.212350 22339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:43.213590 22339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:43.213636 22339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:43.213649 22339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:43.213660 22339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:43.213671 22339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:43.215113 22339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:43.215322 22339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:43.215499 22339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:43.219995 22347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:43.220005 22345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:43.220028 22344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:43.220196 22339 server_base.cc:1034] running on GCE node
I20260928 22:48:43.220422 22339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:43.220705 22339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:43.221922 22339 hybrid_clock.cc:648] HybridClock initialized: now 1790635723221904 us; error 32 us; skew 500 ppm
Sep 28 22:48:43 dist-test-slave-pxr1 krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:43.224362 22339 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:43.224817 22339 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260928 22:48:43.225051 22339 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:43.225097 22339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:43.225219 22339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:43.226302 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "1c30ed73bda04dfca282e3c8be000bab"
format_stamp: "Formatted at 2026-09-28 22:48:43 on dist-test-slave-pxr1"
server_key: "25cde40ea44ba2650fc78f733b4f0d95"
server_key_iv: "0e90f7a2fdbce699eb92d74d511d99fa"
server_key_version: "encryptionkey@0"
I20260928 22:48:43.226631 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c30ed73bda04dfca282e3c8be000bab"
format_stamp: "Formatted at 2026-09-28 22:48:43 on dist-test-slave-pxr1"
server_key: "25cde40ea44ba2650fc78f733b4f0d95"
server_key_iv: "0e90f7a2fdbce699eb92d74d511d99fa"
server_key_version: "encryptionkey@0"
I20260928 22:48:43.227886 22339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:43.228567 22354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:43.228701 22339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:43.228816 22339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "1c30ed73bda04dfca282e3c8be000bab"
format_stamp: "Formatted at 2026-09-28 22:48:43 on dist-test-slave-pxr1"
server_key: "25cde40ea44ba2650fc78f733b4f0d95"
server_key_iv: "0e90f7a2fdbce699eb92d74d511d99fa"
server_key_version: "encryptionkey@0"
I20260928 22:48:43.228876 22339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:43.238997 22339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:43.241132 22339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:43.241266 22339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:43.244817 22339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41555
I20260928 22:48:43.244851 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41555 every 8 connection(s)
I20260928 22:48:43.245177 22339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:43.246012 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:43.246043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22339
I20260928 22:48:43.246151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:43.246312 392 external_mini_cluster.cc:1428] Setting key 0fe7ce248e61884f25eda559116527bf
I20260928 22:48:43.248898 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab: Bootstrap starting.
I20260928 22:48:43.249557 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:43.249879 22407 log.cc:824] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:43.250478 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab: No bootstrap required, opened a new log
I20260928 22:48:43.251488 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:43.246783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39223 (local address 127.0.98.62:41555)
0928 22:48:43.246936 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:43.246944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:43.246968 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:48:43.247394 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:43.247401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:43.247447 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:43.247558 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:43.247843 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:43.248712 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:43.249767 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:43.250200 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:43.250297 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:43.250943 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:43.250948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:43.250948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:43.251098 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:43.251126 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:43.251137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:43.251189 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:43.251259 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 22:48:43.251321 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 22:48:43.252350 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:43.252405 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260928 22:48:43.252548 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:43.252596 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c30ed73bda04dfca282e3c8be000bab, State: Initialized, Role: FOLLOWER
I20260928 22:48:43.252709 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260928 22:48:43.252805 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:43.252841 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:43.252885 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:43.253439 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260928 22:48:43.253532 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c30ed73bda04dfca282e3c8be000bab; no voters:
I20260928 22:48:43.253705 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:43.253736 22416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:43.253858 22416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [term 1 LEADER]: Becoming Leader. State: Replica: 1c30ed73bda04dfca282e3c8be000bab, State: Running, Role: LEADER
I20260928 22:48:43.253942 22416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260928 22:48:43.254050 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:43.254297 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c30ed73bda04dfca282e3c8be000bab. Latest consensus state: current_term: 1 leader_uuid: "1c30ed73bda04dfca282e3c8be000bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } } }
I20260928 22:48:43.254369 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:43.254522 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c30ed73bda04dfca282e3c8be000bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c30ed73bda04dfca282e3c8be000bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } } }
I20260928 22:48:43.254585 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:43.254649 22421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:43.255203 22421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:43.257205 22421 catalog_manager.cc:1347] Generated new cluster ID: 5f56b376152b4db3b4f73bf75f9ef340
I20260928 22:48:43.257304 22421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:43.257983 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:43.253687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59357 (local address 127.0.98.62:41555)
0928 22:48:43.253724 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:43.253725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:43.254922 (+ 1197us) server_negotiation.cc:407] Connection header received
0928 22:48:43.254970 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:43.254971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:43.254982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:43.255041 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:43.255238 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:43.255944 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:43.256808 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:43.257053 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:43.257113 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:43.257729 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:43.257731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:43.257731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:43.257736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:43.257753 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:43.257757 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:43.257816 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:43.257885 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:48:43.257910 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:48:43.266188 22421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:43.266701 22421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:43.269749 22421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c30ed73bda04dfca282e3c8be000bab: Generated new TSK 0
I20260928 22:48:43.269956 22421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:43.277576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22339
2026-09-28T22:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:43 dist-test-slave-pxr1 krb5kdc[22438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:43 dist-test-slave-pxr1 krb5kdc[22438](info): set up 2 sockets
Sep 28 22:48:43 dist-test-slave-pxr1 krb5kdc[22438](info): commencing operation
krb5kdc: starting...
W20260928 22:48:44.232550 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:44 dist-test-slave-pxr1 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:44.341504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:44.420449 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:44.420666 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:44.420689 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:44.421901 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:44.421960 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:44.421974 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:44.421989 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:44.422012 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:44.423578 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:44.423807 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:44.424010 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:44.428788 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:44.428746 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:44.428792 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:44.428934 22451 server_base.cc:1034] running on GCE node
I20260928 22:48:44.429141 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:44.429406 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:44.430546 22451 hybrid_clock.cc:648] HybridClock initialized: now 1790635724430535 us; error 28 us; skew 500 ppm
Sep 28 22:48:44 dist-test-slave-pxr1 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:44.432937 22451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:44.433323 22451 webserver.cc:460] Webserver started at http://127.0.98.62:42535/ using document root <none> and password file <none>
I20260928 22:48:44.433495 22451 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:44.433533 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:44.433629 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:44.434643 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "336a93ef0c844d87bcfc891711df4d24"
format_stamp: "Formatted at 2026-09-28 22:48:44 on dist-test-slave-pxr1"
server_key: "51354a42699f5ac52756e5a7b1622f3f"
server_key_iv: "4ed07b075ec3ad9d0cd19183f7c0886e"
server_key_version: "encryptionkey@0"
I20260928 22:48:44.434938 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "336a93ef0c844d87bcfc891711df4d24"
format_stamp: "Formatted at 2026-09-28 22:48:44 on dist-test-slave-pxr1"
server_key: "51354a42699f5ac52756e5a7b1622f3f"
server_key_iv: "4ed07b075ec3ad9d0cd19183f7c0886e"
server_key_version: "encryptionkey@0"
I20260928 22:48:44.436204 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:44.436940 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:44.437081 22451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:44.437155 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "336a93ef0c844d87bcfc891711df4d24"
format_stamp: "Formatted at 2026-09-28 22:48:44 on dist-test-slave-pxr1"
server_key: "51354a42699f5ac52756e5a7b1622f3f"
server_key_iv: "4ed07b075ec3ad9d0cd19183f7c0886e"
server_key_version: "encryptionkey@0"
I20260928 22:48:44.437211 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:44.441087 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:44.443106 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:44.443217 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:44.446812 22451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39239
I20260928 22:48:44.446848 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39239 every 8 connection(s)
I20260928 22:48:44.447137 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:44.448071 22519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:44.450399 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24: Bootstrap starting.
I20260928 22:48:44.450927 22519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:44.451179 22519 log.cc:824] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:44.451815 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24: No bootstrap required, opened a new log
I20260928 22:48:44.453399 22519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260928 22:48:44.453538 22519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:44.453570 22519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336a93ef0c844d87bcfc891711df4d24, State: Initialized, Role: FOLLOWER
I20260928 22:48:44.453711 22519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260928 22:48:44.453785 22519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:44.453815 22519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:44.453861 22519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:44.454375 22519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260928 22:48:44.454468 22519 leader_election.cc:302] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 336a93ef0c844d87bcfc891711df4d24; no voters:
I20260928 22:48:44.454629 22519 leader_election.cc:288] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:44.454675 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:44.454841 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [term 1 LEADER]: Becoming Leader. State: Replica: 336a93ef0c844d87bcfc891711df4d24, State: Running, Role: LEADER
I20260928 22:48:44.454866 22519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:44.454982 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260928 22:48:44.455327 22523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "336a93ef0c844d87bcfc891711df4d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260928 22:48:44.455335 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 336a93ef0c844d87bcfc891711df4d24. Latest consensus state: current_term: 1 leader_uuid: "336a93ef0c844d87bcfc891711df4d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336a93ef0c844d87bcfc891711df4d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260928 22:48:44.455447 22523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:44.455467 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:44.455658 22526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:44.455826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22451
I20260928 22:48:44.455931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:44.456108 392 external_mini_cluster.cc:1428] Setting key 7b1f606843b570ef0d7ccf8d9b480515
I20260928 22:48:44.456221 22526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:44.458292 22526 catalog_manager.cc:1347] Generated new cluster ID: e718249f0c894638aa37babd86486513
I20260928 22:48:44.458348 22526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:44.458630 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:44.456781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59241 (local address 127.0.98.62:39239)
0928 22:48:44.457158 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:44.457166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:44.457191 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:48:44.457317 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:44.457324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:44.457356 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:44.457495 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:44.458179 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:44.458184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:44.458184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:44.458189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:44.458232 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:44.458248 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:44.458308 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:44.458423 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:48:44.458466 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":75,"threads_started":1}
I20260928 22:48:44.459671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:44.462158 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:44.461188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38221 (local address 127.0.98.62:39239)
0928 22:48:44.461227 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:44.461229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:44.461264 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:48:44.461323 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:44.461324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:44.461329 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:44.461390 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:44.461944 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:44.461945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:44.461945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:48:44.461949 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:44.461966 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:48:44.461970 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:44.462037 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:44.462083 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 22:48:44.462096 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:48:44.477542 22526 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:44.477716 22526 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:44.482959 22526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 336a93ef0c844d87bcfc891711df4d24: Generated new TSK 0
I20260928 22:48:44.483144 22526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:48:44.490289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22451
2026-09-28T22:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1212 ms)
[----------] 18 tests from AuthTokenIssuingTest (21889 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:44 dist-test-slave-pxr1 krb5kdc[22550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:44 dist-test-slave-pxr1 krb5kdc[22550](info): set up 2 sockets
Sep 28 22:48:44 dist-test-slave-pxr1 krb5kdc[22550](info): commencing operation
krb5kdc: starting...
W20260928 22:48:45.432639 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:45.543412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:45.623553 22563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:45.623790 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:45.623816 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:45.623842 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:45.625094 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:45.625140 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:45.625159 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:45.625174 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:45.625191 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:45.626763 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:45.627017 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:45.627231 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:45.631810 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:45.631990 22563 server_base.cc:1034] running on GCE node
W20260928 22:48:45.631835 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:45.631822 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:45.632311 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:45.632604 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:45.633764 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790635725633771 us; error 50 us; skew 500 ppm
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:45.636183 22563 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:45.636633 22563 webserver.cc:460] Webserver started at http://127.0.98.62:44801/ using document root <none> and password file <none>
I20260928 22:48:45.636884 22563 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:45.636935 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:45.637058 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:45.638207 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "86707eb48e6a4599b36b1011165fbb5b"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "d11bbd2f51d5372d67a10def0cbc15fd"
server_key_iv: "9ffa4b9ee080f4679c6365f84336fede"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.638556 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "86707eb48e6a4599b36b1011165fbb5b"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "d11bbd2f51d5372d67a10def0cbc15fd"
server_key_iv: "9ffa4b9ee080f4679c6365f84336fede"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.639845 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:45.640525 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:45.640656 22563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:45.640760 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "86707eb48e6a4599b36b1011165fbb5b"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "d11bbd2f51d5372d67a10def0cbc15fd"
server_key_iv: "9ffa4b9ee080f4679c6365f84336fede"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.640825 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:45.653071 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:45.655293 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:45.655437 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:45.659379 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41817
I20260928 22:48:45.659427 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41817 every 8 connection(s)
I20260928 22:48:45.659750 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:45.660689 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:45.662281 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.668022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22563
I20260928 22:48:45.668143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:45.668329 392 external_mini_cluster.cc:1428] Setting key fb3197057bff1d074d8b27c526963fd7
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:45.671818 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.663000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.98.62:41817)
0928 22:48:45.663168 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.663176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.663250 (+ 74us) server_negotiation.cc:407] Connection header received
0928 22:48:45.663821 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.663823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.663840 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.663920 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.664420 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.665295 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.666162 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.666482 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.666560 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.668826 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.668833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.668834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.668841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.670367 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.670721 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.670724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.670725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.670802 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.670991 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.670991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.670992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.671207 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.671276 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.671445 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.671570 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260928 22:48:45.671824 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.662958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41817 (local address 127.0.0.1:43284)
0928 22:48:45.663140 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.663181 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.663668 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.663940 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.663942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.664185 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.664343 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.664346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.665384 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.665385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.666087 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.666089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.666558 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.666558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.666610 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.667108 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.667116 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.668745 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.670451 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.670456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.670459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.670651 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.670858 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.670858 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.670859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.670940 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.671266 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.671270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.671336 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.671422 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.671638 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":73,"thread_start_us":76,"threads_started":1}
I20260928 22:48:45.672552 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.672935 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.672825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49016)
0928 22:48:45.672865 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.672911 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 22:48:45.673044 22580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
I20260928 22:48:45.673452 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.673362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49024)
0928 22:48:45.673405 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.673429 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 22:48:45.673583 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43791: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 22:48:45.689639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43791
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:48:45.722855 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } attempt: 1
I20260928 22:48:45.723359 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.723213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49038)
0928 22:48:45.723275 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.723324 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:48:45.723810 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.723692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49042)
0928 22:48:45.723750 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.723783 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 22:48:45.723935 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43791: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
W20260928 22:48:45.768863 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:45.769090 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:45.769107 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:45.769129 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:45.770354 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:45.770406 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:45.770418 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:45.770430 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:45.770440 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:45.771958 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:45.772194 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:45.772418 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:45.777011 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:45.777006 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:45.777038 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:45.777207 22638 server_base.cc:1034] running on GCE node
I20260928 22:48:45.777503 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:45.777771 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:45.778916 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790635725778903 us; error 29 us; skew 500 ppm
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:45.781371 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 22:48:45.781840 22638 webserver.cc:460] Webserver started at http://127.0.98.61:41467/ using document root <none> and password file <none>
I20260928 22:48:45.782037 22638 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:45.782083 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:45.782207 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:45.783248 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data/instance:
uuid: "85c1def0f0fa4da796e9db75cf92a7fc"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "2d99510398a0a9e4ae5130c116e5f68c"
server_key_iv: "fc6e7a3c39abe59de2d170a1f413bff1"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.783581 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal/instance:
uuid: "85c1def0f0fa4da796e9db75cf92a7fc"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "2d99510398a0a9e4ae5130c116e5f68c"
server_key_iv: "fc6e7a3c39abe59de2d170a1f413bff1"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.784883 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:45.785648 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:45.785820 22638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:45.785902 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "85c1def0f0fa4da796e9db75cf92a7fc"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "2d99510398a0a9e4ae5130c116e5f68c"
server_key_iv: "fc6e7a3c39abe59de2d170a1f413bff1"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.785974 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:45.792105 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } attempt: 2
I20260928 22:48:45.792547 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.792406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49046)
0928 22:48:45.792463 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.792509 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:48:45.792886 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.792813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49050)
0928 22:48:45.792845 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.792868 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 22:48:45.792968 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43791: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43791: connect: Connection refused (error 111)
I20260928 22:48:45.808480 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:45.810360 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:45.810478 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:45.814302 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43791
I20260928 22:48:45.814339 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43791 every 8 connection(s)
I20260928 22:48:45.814662 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
I20260928 22:48:45.815559 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:45.817130 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:45.824388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22638
I20260928 22:48:45.824496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-1/wal/instance
I20260928 22:48:45.824666 392 external_mini_cluster.cc:1428] Setting key 07b37b29b28a83ce847b1aeb3ccfdca6
I20260928 22:48:45.825052 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.817863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.0.98.62:41817)
0928 22:48:45.817917 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.817918 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.818106 (+ 188us) server_negotiation.cc:407] Connection header received
0928 22:48:45.818564 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.818565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.818577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.818624 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.819153 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.819627 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.820511 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.820693 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.820797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.822697 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.822698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.822698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.822703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.823794 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.824126 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.824127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.824127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.824171 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.824431 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.824431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.824432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.824550 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.824602 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.824918 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.824987 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:48:45.825273 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.817853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41817 (local address 127.0.0.1:43296)
0928 22:48:45.818007 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.818041 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.818489 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.818678 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.818682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.818913 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.819093 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.819097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.819685 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.819686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.820454 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.820456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.820819 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.820820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.820911 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.821362 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.821370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.822625 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.823857 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.823862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.823864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.824071 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.824231 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.824232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.824232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.824362 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.824597 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.824605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.824786 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.824902 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.825057 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 22:48:45.825820 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 22:48:45.832304 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.826094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49054)
0928 22:48:45.826142 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.826153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.826193 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.826384 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.826384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.826512 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.826555 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.826556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.827324 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.827325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.827854 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.827856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.828276 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.828276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.828335 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.828789 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.828794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.829868 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.831384 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.831385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.831385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.831553 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.831719 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.831720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.831720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.831772 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.832010 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.832011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.832048 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.832109 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.832209 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:48:45.832407 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.826135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.0.98.61:43791)
0928 22:48:45.826274 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.826278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.826291 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:48:45.826306 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.826308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.826317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.826374 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.826613 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.827257 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.827929 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.828215 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.828271 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.829944 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.829952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.829953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.829959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.831312 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.831620 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.831622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.831623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.831667 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.831818 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.831819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.831820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.831952 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.832020 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.832144 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.832292 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260928 22:48:45.832904 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.833235 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.833134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59470)
0928 22:48:45.833173 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.833214 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 22:48:45.833359 22657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 22:48:45.834005 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.833916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59482)
0928 22:48:45.833957 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.833983 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:48:45.834106 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 22:48:45.845532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41055
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:48:45.883352 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } attempt: 1
I20260928 22:48:45.883831 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.883711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59498)
0928 22:48:45.883770 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.883801 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:48:45.884130 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } attempt: 3
I20260928 22:48:45.884303 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.884188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59512)
0928 22:48:45.884238 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.884273 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:48:45.884485 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:48:45.890770 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.884479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.61:43791)
0928 22:48:45.884513 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.884514 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.884523 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:45.884551 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.884551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.884562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.884629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.884985 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.885476 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.886153 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.886366 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.886421 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.888467 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.888468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.888469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.888475 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.889821 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.890175 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.890176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.890176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.890222 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.890367 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.890368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.890368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.890464 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.890506 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.890643 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.890706 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:45.890818 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.884398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49068)
0928 22:48:45.884446 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.884456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.884499 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.884635 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.884636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.884843 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.884910 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.884912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.885559 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.885561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.886080 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.886081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.886426 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.886427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.886506 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.886959 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.886963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.888379 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.889896 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.889897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.889898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.890110 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.890260 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.890261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.890261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.890319 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.890511 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.890512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.890563 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.890626 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.890723 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:48:45.891330 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.891664 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.891566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59516)
0928 22:48:45.891610 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.891641 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:48:45.891958 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.891875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59520)
0928 22:48:45.891914 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.891942 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:48:45.892041 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
I20260928 22:48:45.923202 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } attempt: 1
I20260928 22:48:45.923679 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.923535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59528)
0928 22:48:45.923594 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.923641 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:48:45.924001 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.923911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59544)
0928 22:48:45.923952 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.923980 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:48:45.924088 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
W20260928 22:48:45.927986 22713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:45.928203 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:45.928233 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:45.928263 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:45.930007 22713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:45.930063 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:45.930083 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:45.930106 22713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:45.930122 22713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:45.931711 22713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41817,127.0.98.61:43791,127.0.98.60:41055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:45.931970 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:45.932192 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:45.936704 22721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:45.936689 22718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:45.936755 22719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:45.936939 22713 server_base.cc:1034] running on GCE node
I20260928 22:48:45.937202 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:45.937482 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:45.938624 22713 hybrid_clock.cc:648] HybridClock initialized: now 1790635725938608 us; error 27 us; skew 500 ppm
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:45.940949 22713 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 22:48:45.941366 22713 webserver.cc:460] Webserver started at http://127.0.98.60:46071/ using document root <none> and password file <none>
I20260928 22:48:45.941572 22713 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:45.941619 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:45.941744 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:45.942766 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data/instance:
uuid: "5d01a71ce58c4c5b80f533473de1183c"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "da124ca578c95be6e84a7452789a04fb"
server_key_iv: "795e046065fd5453d14cc2c37bac4ee8"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.943131 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal/instance:
uuid: "5d01a71ce58c4c5b80f533473de1183c"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "da124ca578c95be6e84a7452789a04fb"
server_key_iv: "795e046065fd5453d14cc2c37bac4ee8"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.944408 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:45.945137 22728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:45.945284 22713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:45.945362 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "5d01a71ce58c4c5b80f533473de1183c"
format_stamp: "Formatted at 2026-09-28 22:48:45 on dist-test-slave-pxr1"
server_key: "da124ca578c95be6e84a7452789a04fb"
server_key_iv: "795e046065fd5453d14cc2c37bac4ee8"
server_key_version: "encryptionkey@0"
I20260928 22:48:45.945422 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:45.952636 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } attempt: 2
I20260928 22:48:45.953084 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.952970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59558)
0928 22:48:45.953023 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.953055 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:48:45.953382 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.953299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59568)
0928 22:48:45.953336 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.953360 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 22:48:45.953462 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41055: connect: Connection refused (error 111)
I20260928 22:48:45.971558 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:45.973316 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:45.973452 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:45.977088 22713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41055
I20260928 22:48:45.977119 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41055 every 8 connection(s)
I20260928 22:48:45.977442 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
I20260928 22:48:45.978334 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:45.979897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22713
I20260928 22:48:45.979983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635468897675-392-0/minicluster-data/master-2/wal/instance
I20260928 22:48:45.979872 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.980154 392 external_mini_cluster.cc:1428] Setting key f038668f52e371ccc2605e7852b02ed1
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:48:45.988503 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.980601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.98.62:41817)
0928 22:48:45.980655 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.980657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.980868 (+ 211us) server_negotiation.cc:407] Connection header received
0928 22:48:45.981355 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.981356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.981369 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.981429 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.981949 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.982446 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.983297 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.983502 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.983557 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.985759 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.985761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.985762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.985770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.987194 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.987580 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.987582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.987582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.987639 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.987843 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.987844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.987844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.987963 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.988009 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.988322 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.988410 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:48:45.988698 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.980577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41817 (local address 127.0.0.1:43304)
0928 22:48:45.980740 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.980781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.981266 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.981446 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.981448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.981703 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.981857 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.981861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.982517 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.982518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.983219 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.983220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.983587 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.983588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.983691 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.984130 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.984137 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.985676 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.987262 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.987269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.987270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.987509 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.987700 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.987701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.987701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.987790 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.988020 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.988026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.988186 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.988289 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.988433 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 22:48:45.988888 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.980576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.0.98.60:41055)
0928 22:48:45.980788 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.980793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.980846 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:48:45.981277 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.981284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.981302 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.981380 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.981979 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.982802 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.983505 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.983809 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.983866 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.986474 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.986481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.986481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.986487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.987801 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.988116 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.988119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.988119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.988177 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.988352 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.988352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.988353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.988481 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.988602 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.988684 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.988780 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260928 22:48:45.989287 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:45.995676 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.989525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43791 (local address 127.0.0.1:49074)
0928 22:48:45.989566 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.989575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.989650 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.989781 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.989781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.989924 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.989964 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.989965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.990574 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.990575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.991054 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.991056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.991394 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.991395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.991455 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.991857 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.991861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:45.993079 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:45.994563 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.994564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.994564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.994767 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.995019 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:45.995020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:45.995020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:45.995078 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:45.995371 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:45.995372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:45.995420 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:48:45.995498 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:48:45.995588 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 22:48:45.995675 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.989590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.0.98.61:43791)
0928 22:48:45.989630 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.989632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.989640 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:45.989719 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.989720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.989730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.989779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.990046 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.990507 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.991122 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.991336 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.991389 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.993162 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.993163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.993163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.993170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.994469 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.994830 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.994832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.994832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.994923 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.995154 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.995155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.995155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.995314 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.995379 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.995516 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.995585 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:48:45.996253 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.989822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.62:41817)
0928 22:48:45.989861 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.989862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.989877 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:48:45.989938 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.989939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.989948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.989990 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.990391 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.990824 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.991445 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.991670 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.991727 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.993694 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:45.993695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:45.993696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:45.993701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:45.995260 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.995585 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.995586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.995586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.995640 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:45.995820 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:45.995820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:45.995821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:45.995923 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:45.996054 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:45.996103 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 22:48:45.996172 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:45.996441 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:45.999215 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } attempt: 2
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:48:46.002892 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.996685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59578)
0928 22:48:45.996778 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.996786 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.996829 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.996931 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.996932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.997053 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:45.997079 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.997080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.997665 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.997666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.998134 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:45.998136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.998461 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.998461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:45.998504 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:45.998862 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:45.998867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:46.000173 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:46.002017 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.002018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.002018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.002166 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.002339 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.002340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.002340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.002388 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.002632 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:46.002633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:46.002666 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 22:48:46.002706 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:48:46.002789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 22:48:46.003175 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.996737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.0.98.60:41055)
0928 22:48:45.996777 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.996778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.996815 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:48:45.996868 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.996869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.996879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.996929 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.997137 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.997600 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.998196 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.998405 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.998457 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.000241 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.000242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.000242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.000250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.001944 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.002223 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.002224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.002224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.002272 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.002459 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.002459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.002460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.002568 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.002994 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.003031 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.003094 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 22:48:45 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:48:46.004173 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.996894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.98.61:43791)
0928 22:48:45.996936 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.996937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.996945 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:45.996974 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.996975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.996983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.997023 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:45.997439 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.997882 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.998506 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:45.998708 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:45.998760 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.001213 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.001215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.001215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.001222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.002909 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.003275 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.003277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.003277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.003334 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.003529 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.003529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.003530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.003663 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.003779 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.003948 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.004019 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:48:46.004889 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: Bootstrap starting.
I20260928 22:48:46.005537 22781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:46.005669 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:46.005824 22781 log.cc:824] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:46.006705 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: No bootstrap required, opened a new log
I20260928 22:48:46.006747 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.999472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59584)
0928 22:48:45.999515 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:45.999526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:45.999573 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:45.999815 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:45.999816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:45.999979 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:46.000023 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:46.000025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.000866 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.000867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:46.001565 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:46.001568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.002317 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.002318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:46.002385 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.002695 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:46.002700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:46.003884 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:46.005716 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.005718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.005718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.005883 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.006090 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.006092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.006092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.006155 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.006441 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:46.006442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:46.006489 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:48:46.006548 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:48:46.006646 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:48:46.006819 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:45.999499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.98.60:41055)
0928 22:48:45.999683 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:45.999685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:45.999694 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:45.999714 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:45.999714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:45.999724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:45.999793 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.000099 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.000806 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.001642 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.002245 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.002305 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.003969 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.003970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.003970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.003976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.005654 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.005944 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.005945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.005946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.006015 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.006227 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.006227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.006228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.006372 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.006434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.006580 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.006645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
I20260928 22:48:46.006805 22605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "86707eb48e6a4599b36b1011165fbb5b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43316
I20260928 22:48:46.009186 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: Bootstrap starting.
I20260928 22:48:46.008563 22781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.009639 22781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d01a71ce58c4c5b80f533473de1183c, State: Initialized, Role: FOLLOWER
I20260928 22:48:46.009753 22781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.009862 22631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:46.010006 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.010070 22781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:46.010098 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.010164 22631 log.cc:824] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:46.010937 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: No bootstrap required, opened a new log
W20260928 22:48:46.011782 22800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:46.011880 22800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:46.012272 22631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.012400 22631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86707eb48e6a4599b36b1011165fbb5b, State: Initialized, Role: FOLLOWER
I20260928 22:48:46.012471 22631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.012784 22631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:46.012815 22605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:48:46.012856 22605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:46.013628 22605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.013865 22605 leader_election.cc:288] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [CANDIDATE]: Term 1 election: Requested vote from peers 85c1def0f0fa4da796e9db75cf92a7fc (127.0.98.61:43791), 5d01a71ce58c4c5b80f533473de1183c (127.0.98.60:41055)
I20260928 22:48:46.011983 22713 master_runner.cc:186] Error getting master registration for 127.0.98.62:41817: OK, instance_id { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" instance_seqno: 1790635725658265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:48:46.014139 22713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:48:46.014326 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86707eb48e6a4599b36b1011165fbb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d01a71ce58c4c5b80f533473de1183c"
I20260928 22:48:46.014439 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [term 0 FOLLOWER]: Advancing to term 1
W20260928 22:48:46.014798 22813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:46.014839 22813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:46.015264 22755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86707eb48e6a4599b36b1011165fbb5b in term 1.
I20260928 22:48:46.015777 22582 leader_election.cc:302] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d01a71ce58c4c5b80f533473de1183c, 86707eb48e6a4599b36b1011165fbb5b; no voters:
I20260928 22:48:46.012710 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.016636 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.016757 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:46.016848 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [term 1 LEADER]: Becoming Leader. State: Replica: 86707eb48e6a4599b36b1011165fbb5b, State: Running, Role: LEADER
I20260928 22:48:46.016961 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.017463 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86707eb48e6a4599b36b1011165fbb5b. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.017588 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:46.017813 22821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:46.018031 22821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:46.016657 22563 master_runner.cc:186] Error getting master registration for 127.0.98.61:43791: OK, instance_id { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" instance_seqno: 1790635725813238 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:48:46.018946 22563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:48:46.020215 22755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [term 1 FOLLOWER]: Refusing update from remote peer 86707eb48e6a4599b36b1011165fbb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:46.020934 22802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:46.022331 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86707eb48e6a4599b36b1011165fbb5b. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.022445 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.032249 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:46.021166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.0.98.61:43791)
0928 22:48:46.021207 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.021209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.021554 (+ 345us) server_negotiation.cc:407] Connection header received
0928 22:48:46.021715 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.021716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.021727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.021759 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.023013 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.023756 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.024819 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.025139 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.025222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.028703 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.028706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.028706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.028754 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.030427 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.031133 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.031134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.031135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.031191 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.031486 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.031487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.031487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.031629 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.031708 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.032064 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.032148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:48:46.036427 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:46.021025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.98.62:41817)
0928 22:48:46.021072 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.021074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.023102 (+ 2028us) server_negotiation.cc:407] Connection header received
0928 22:48:46.023193 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.023195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.023203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.023231 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.025864 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.026976 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.029602 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.029905 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.029948 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.031324 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.031326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.031326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.031334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.033786 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.035284 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.035286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.035286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.035351 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.035799 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.035801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.035801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.035943 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.036027 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.036222 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.036310 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:48:46.038120 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:46.021646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.0.98.60:41055)
0928 22:48:46.021682 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.021683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.026067 (+ 4384us) server_negotiation.cc:407] Connection header received
0928 22:48:46.026158 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.026159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.026172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.026199 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.029414 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.030360 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.031318 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.031620 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.031662 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.034715 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.034716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.034717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.034725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.036329 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.037014 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.037015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.037015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.037074 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.037379 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.037380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.037381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.037543 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.037639 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.037926 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.038019 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:46.042136 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.042241 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:46.042425 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.042523 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.042714 22826 mvcc.cc:205] Tried to move back new op lower bound from 7334443933776859136 to 7334443933766795264. Current Snapshot: MvccSnapshot[applied={T|T < 7334443933776859136}]
I20260928 22:48:46.043347 22821 catalog_manager.cc:1347] Generated new cluster ID: 72b40d90c95a4090875ac7942c054c7f
I20260928 22:48:46.043437 22821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:46.044530 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } attempt: 3
Sep 28 22:48:46 dist-test-slave-pxr1 krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:48:46.052919 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:46.044845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41055 (local address 127.0.0.1:59614)
0928 22:48:46.044900 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:46.044911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:46.044953 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:46.045085 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:46.045086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:46.045258 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:46.045311 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:46.045313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.046218 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.046219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:46.047083 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:46.047087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.047536 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.047537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:46.047595 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.048032 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:46.048038 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:46.049684 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:46.051700 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.051702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.051702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.051922 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.052155 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:46.052155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:46.052156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:46.052227 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:46.052557 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:46.052558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:46.052619 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 22:48:46.052677 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:48:46.052811 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:48:46.053314 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:46.044880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.0.98.60:41055)
0928 22:48:46.044928 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.044930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.044938 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:48:46.045008 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.045009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.045018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.045067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.045405 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.046139 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.047159 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:46.047415 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:46.047477 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:46.049793 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:46.049794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:46.049794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:46.049802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:46.051592 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.052007 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.052009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.052009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.052082 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:46.052301 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:46.052302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:46.052302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:46.052457 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:46.052515 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:46.053138 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 22:48:46.053224 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:48:46.054901 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: Bootstrap starting.
I20260928 22:48:46.055642 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:46.055938 22706 log.cc:824] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:46.065500 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: No bootstrap required, opened a new log
I20260928 22:48:46.067553 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.067718 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c1def0f0fa4da796e9db75cf92a7fc, State: Initialized, Role: FOLLOWER
I20260928 22:48:46.067812 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } }
I20260928 22:48:46.068173 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:46.068840 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:46.069800 22679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [term 1 FOLLOWER]: Refusing update from remote peer 86707eb48e6a4599b36b1011165fbb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:46.070174 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86707eb48e6a4599b36b1011165fbb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85c1def0f0fa4da796e9db75cf92a7fc"
I20260928 22:48:46.070169 22818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:46.070995 22680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86707eb48e6a4599b36b1011165fbb5b in term 1.
I20260928 22:48:46.074821 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.074932 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.074982 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86707eb48e6a4599b36b1011165fbb5b. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.075021 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.075045 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.075078 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:46.075531 22835 mvcc.cc:205] Tried to move back new op lower bound from 7334443933776859136 to 7334443933766795264. Current Snapshot: MvccSnapshot[applied={T|T < 7334443933776859136}]
W20260928 22:48:46.077325 22846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:46.077387 22846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:46.088881 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86707eb48e6a4599b36b1011165fbb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86707eb48e6a4599b36b1011165fbb5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41817 } } peers { permanent_uuid: "85c1def0f0fa4da796e9db75cf92a7fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43791 } } peers { permanent_uuid: "5d01a71ce58c4c5b80f533473de1183c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41055 } } }
I20260928 22:48:46.089010 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:46.091574 22821 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:46.092360 22821 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:46.113018 22821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86707eb48e6a4599b36b1011165fbb5b: Generated new TSK 0
I20260928 22:48:46.113240 22821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:48:46.136868 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.135429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.0.98.60:41055)
0928 22:48:46.135477 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.135479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.136141 (+ 662us) server_negotiation.cc:407] Connection header received
0928 22:48:46.136246 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.136247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.136259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.136288 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.136790 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59616: BlockingRecv error: recv got EOF from 127.0.0.1:59616 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:48:46.137635 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.135868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.98.61:43791)
0928 22:48:46.135908 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.135909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.137110 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 22:48:46.137203 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.137204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.137215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.137243 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.137558 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49106: BlockingRecv error: recv got EOF from 127.0.0.1:49106 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:48:46.138628 22634 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.137980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.0.98.62:41817)
0928 22:48:46.138018 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.138020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.138133 (+ 113us) server_negotiation.cc:407] Connection header received
0928 22:48:46.138223 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.138224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.138234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.138260 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.138564 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:48:46.141065 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.140236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.0.98.61:43791)
0928 22:48:46.140282 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.140284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.140386 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:48:46.140587 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.140589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.140610 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.140639 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.141013 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49120: BlockingRecv error: recv got EOF from 127.0.0.1:49120 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 22:48:46.144384 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.143499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.98.61:43791)
0928 22:48:46.143547 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.143549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.143792 (+ 243us) server_negotiation.cc:407] Connection header received
0928 22:48:46.143885 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.143886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.143896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.143924 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.144329 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49130: BlockingRecv error: recv got EOF from 127.0.0.1:49130 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 22:48:46.148526 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.147636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.98.61:43791)
0928 22:48:46.147687 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.147689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.147927 (+ 238us) server_negotiation.cc:407] Connection header received
0928 22:48:46.148093 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.148094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.148105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.148132 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.148465 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49136: BlockingRecv error: recv got EOF from 127.0.0.1:49136 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 22:48:46.154834 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.153886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.98.61:43791)
0928 22:48:46.153930 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.153933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.154245 (+ 312us) server_negotiation.cc:407] Connection header received
0928 22:48:46.154346 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.154347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.154359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.154388 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.154772 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49140: BlockingRecv error: recv got EOF from 127.0.0.1:49140 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:48:46.165210 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.164220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.0.98.61:43791)
0928 22:48:46.164263 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.164265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.164565 (+ 300us) server_negotiation.cc:407] Connection header received
0928 22:48:46.164662 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.164664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.164676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.164705 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.165131 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49152: BlockingRecv error: recv got EOF from 127.0.0.1:49152 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:48:46.183429 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.182569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.0.98.61:43791)
0928 22:48:46.182611 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.182613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.182861 (+ 248us) server_negotiation.cc:407] Connection header received
0928 22:48:46.182954 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.182955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.182966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.182992 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.183368 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49160: BlockingRecv error: recv got EOF from 127.0.0.1:49160 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:48:46.217345 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.216701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.0.98.61:43791)
0928 22:48:46.216782 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.216784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.216831 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:48:46.216903 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.216904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.216916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.216971 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.217283 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49168: BlockingRecv error: recv got EOF from 127.0.0.1:49168 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 22:48:46.283139 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.282510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.0.98.61:43791)
0928 22:48:46.282573 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.282575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.282605 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:48:46.282688 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.282689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.282701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.282743 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.283077 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49184: BlockingRecv error: recv got EOF from 127.0.0.1:49184 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 22:48:46.412868 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.412207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.98.61:43791)
0928 22:48:46.412260 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.412262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.412303 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:48:46.412383 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.412384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.412394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.412437 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.412808 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49200: BlockingRecv error: recv got EOF from 127.0.0.1:49200 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 22:48:46.670662 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:46.670081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.98.61:43791)
0928 22:48:46.670138 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:46.670140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:46.670151 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:48:46.670213 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:46.670214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:46.670227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:46.670272 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:46.670602 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49208: BlockingRecv error: recv got EOF from 127.0.0.1:49208 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:48:47.012356 22800 catalog_manager.cc:1259] Loaded cluster ID: 72b40d90c95a4090875ac7942c054c7f
I20260928 22:48:47.012431 22800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: loading cluster ID for follower catalog manager: success
I20260928 22:48:47.013396 22800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: acquiring CA information for follower catalog manager: success
I20260928 22:48:47.013681 22800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d01a71ce58c4c5b80f533473de1183c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:48:47.077795 22846 catalog_manager.cc:1259] Loaded cluster ID: 72b40d90c95a4090875ac7942c054c7f
I20260928 22:48:47.077867 22846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: loading cluster ID for follower catalog manager: success
I20260928 22:48:47.078826 22846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: acquiring CA information for follower catalog manager: success
I20260928 22:48:47.079144 22846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85c1def0f0fa4da796e9db75cf92a7fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:48:47.186030 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:47.183723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.98.61:43791)
0928 22:48:47.183848 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:47.183849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:47.183859 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:48:47.183918 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:47.183918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:47.183966 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:47.184032 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:47.184150 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.184627 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.185376 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.185570 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.185617 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:47.185732 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:47.185845 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:47.185897 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:47.185927 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 22:48:47.185945 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 22:48:47.190027 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:47.187757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.0.98.60:41055)
0928 22:48:47.187858 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:47.187860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:47.187867 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:48:47.187901 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:47.187902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:47.187935 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:47.188002 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:47.188077 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.188553 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.189331 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.189521 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.189569 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:47.189739 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:47.189854 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:47.189912 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:47.189929 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 22:48:47.189945 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
I20260928 22:48:47.194450 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:47.191648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.0.98.60:41055)
0928 22:48:47.191682 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:47.191684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:47.191856 (+ 172us) server_negotiation.cc:407] Connection header received
0928 22:48:47.191937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:47.191939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:47.191947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:47.191991 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:47.192106 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.192840 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.193806 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.193985 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.194046 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:47.194208 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:47.194244 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:47.194298 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:47.194351 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:48:47.194373 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:48:47.195075 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:47.191599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.62:41817)
0928 22:48:47.191739 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:47.191741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:47.191910 (+ 169us) server_negotiation.cc:407] Connection header received
0928 22:48:47.191956 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:47.191957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:47.192003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:47.192075 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:47.192213 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.192710 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.194083 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.194376 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.194429 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:47.194652 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:47.194761 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:47.194832 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:47.194967 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:48:47.194998 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260928 22:48:47.195297 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:47.191567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.98.61:43791)
0928 22:48:47.191605 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:47.191607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:47.191812 (+ 205us) server_negotiation.cc:407] Connection header received
0928 22:48:47.191872 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:47.191873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:47.191884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:47.191922 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:47.192050 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.192831 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.194616 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:47.194824 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:47.194883 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:47.195045 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:47.195076 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:47.195123 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:47.195209 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:48:47.195234 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:47.196652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22563
I20260928 22:48:47.201346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22638
I20260928 22:48:47.205526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22713
2026-09-28T22:48:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2716 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:47 dist-test-slave-pxr1 krb5kdc[22931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:47 dist-test-slave-pxr1 krb5kdc[22931](info): set up 2 sockets
Sep 28 22:48:47 dist-test-slave-pxr1 krb5kdc[22931](info): commencing operation
krb5kdc: starting...
W20260928 22:48:48.169050 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/test-user.keytab.
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 22:48:48.281188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:48.362258 22944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:48.362493 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:48.362519 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:48.362545 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:48.363727 22944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:48.363775 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:48.363794 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:48.363811 22944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:48.363826 22944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:48.365381 22944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:48.365644 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:48.365855 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:48.370517 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:48.370520 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:48.370494 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:48.370776 22944 server_base.cc:1034] running on GCE node
I20260928 22:48:48.371002 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:48.371284 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:48.372427 22944 hybrid_clock.cc:648] HybridClock initialized: now 1790635728372398 us; error 47 us; skew 500 ppm
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:48.374878 22944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 22:48:48.375330 22944 webserver.cc:460] Webserver started at http://127.0.98.62:33123/ using document root <none> and password file <none>
I20260928 22:48:48.375523 22944 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:48.375571 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:48.375694 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:48.376744 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data/instance:
uuid: "a8dfa536e0334c8ca45a754cd44ed753"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "ef0f0f79db2e192bf5bc663ea7e97b9a"
server_key_iv: "5467a8564bd3c4785525e25f71ecb6be"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.377107 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8dfa536e0334c8ca45a754cd44ed753"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "ef0f0f79db2e192bf5bc663ea7e97b9a"
server_key_iv: "5467a8564bd3c4785525e25f71ecb6be"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.378439 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:48.379252 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:48.379411 22944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:48.379498 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal
uuid: "a8dfa536e0334c8ca45a754cd44ed753"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "ef0f0f79db2e192bf5bc663ea7e97b9a"
server_key_iv: "5467a8564bd3c4785525e25f71ecb6be"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.379570 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:48.406368 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:48.408614 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:48.408795 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:48.412676 22944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34103
I20260928 22:48:48.412757 23011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34103 every 8 connection(s)
I20260928 22:48:48.413102 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:48.414075 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:48.415665 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:48.416203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 22944
I20260928 22:48:48.416304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-0/wal/instance
I20260928 22:48:48.416605 392 external_mini_cluster.cc:1428] Setting key c5252553f1043301df964c148dc351b0
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 22:48:48.426154 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.416388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.0.1:34916)
0928 22:48:48.416629 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.416679 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.417334 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.417610 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.417612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.417950 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.418192 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.418200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.419372 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.419373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.420066 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.420069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.420811 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.420812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.420878 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.421492 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.421501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.423007 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.424871 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.424878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.424880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.425085 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.425286 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.425287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.425287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.425367 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.425669 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.425673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.425738 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.425803 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.426013 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":69,"thread_start_us":91,"threads_started":1}
I20260928 22:48:48.426159 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.416422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.98.62:34103)
0928 22:48:48.416626 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.416634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.416794 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:48:48.417487 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.417489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.417507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.417584 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.418303 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.419259 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.420172 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.420662 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.420814 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.423099 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.423106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.423106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.423113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.424796 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.425145 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.425148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.425148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.425233 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.425415 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.425416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.425416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.425621 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.425679 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.425850 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.425942 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 22:48:48.426893 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:48.427227 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.427100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45398)
0928 22:48:48.427146 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.427201 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:48:48.427398 22962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
I20260928 22:48:48.427814 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.427727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45408)
0928 22:48:48.427768 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.427794 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:48:48.427913 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 22:48:48.440979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43707
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:48:48.477149 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } attempt: 1
I20260928 22:48:48.477703 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.477561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45416)
0928 22:48:48.477626 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.477672 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:48:48.478037 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.477955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45426)
0928 22:48:48.477994 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.478020 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:48:48.478138 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
W20260928 22:48:48.521596 23019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:48.521822 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:48.521847 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:48.521872 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:48.523038 23019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:48.523087 23019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:48.523103 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:48.523119 23019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:48.523133 23019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:48.524672 23019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:48.524948 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:48.525170 23019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:48.529807 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:48.529819 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:48.529826 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:48.530153 23019 server_base.cc:1034] running on GCE node
I20260928 22:48:48.530391 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:48.530654 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:48.531803 23019 hybrid_clock.cc:648] HybridClock initialized: now 1790635728531789 us; error 25 us; skew 500 ppm
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:48.534178 23019 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 22:48:48.534602 23019 webserver.cc:460] Webserver started at http://127.0.98.61:39701/ using document root <none> and password file <none>
I20260928 22:48:48.534777 23019 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:48.534816 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:48.534916 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:48.535939 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data/instance:
uuid: "0e18849b79bb4c08a578470f5cc260c5"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "526bd84b5d785496c57f80f9c2a11fec"
server_key_iv: "60b8ed4caa45ed2ee851166fe2d57312"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.536233 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal/instance:
uuid: "0e18849b79bb4c08a578470f5cc260c5"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "526bd84b5d785496c57f80f9c2a11fec"
server_key_iv: "60b8ed4caa45ed2ee851166fe2d57312"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.537508 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:48.538178 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:48.538350 23019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:48.538434 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal
uuid: "0e18849b79bb4c08a578470f5cc260c5"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "526bd84b5d785496c57f80f9c2a11fec"
server_key_iv: "60b8ed4caa45ed2ee851166fe2d57312"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.538493 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:48.546289 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } attempt: 2
I20260928 22:48:48.546768 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.546638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45432)
0928 22:48:48.546695 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.546740 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:48:48.547052 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.546983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45440)
0928 22:48:48.547014 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.547037 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 22:48:48.547132 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43707: connect: Connection refused (error 111)
I20260928 22:48:48.553710 23019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:48.555667 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:48.555797 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:48.559679 23019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43707
I20260928 22:48:48.559715 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43707 every 8 connection(s)
I20260928 22:48:48.560037 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/data/info.pb
I20260928 22:48:48.560992 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:48.562522 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:48.565609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 23019
I20260928 22:48:48.565703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-1/wal/instance
I20260928 22:48:48.565881 392 external_mini_cluster.cc:1428] Setting key 7841f26177527ebcef55aad3e88b35c6
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:48.571096 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.563233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.0.98.62:34103)
0928 22:48:48.563299 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.563301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.563516 (+ 215us) server_negotiation.cc:407] Connection header received
0928 22:48:48.563955 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.563956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.563968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.564024 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.564598 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.565118 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.566140 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.566416 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.566494 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.568447 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.568449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.568449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.568457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.569763 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.570129 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.570130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.570130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.570185 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.570451 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.570452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.570452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.570575 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.570635 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.570959 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.571028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:48:48.571357 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.563184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.0.1:34930)
0928 22:48:48.563353 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.563389 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.563876 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.564089 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.564093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.564342 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.564534 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.564539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.565182 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.565183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.566062 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.566065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.566502 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.566503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.566603 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.567077 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.567086 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.568357 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.569835 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.569843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.569846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.570059 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.570242 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.570243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.570243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.570377 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.570641 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.570648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.570822 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.570944 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.571120 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 22:48:48.572003 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:48:48.579432 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.572264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45442)
0928 22:48:48.572314 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.572325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.572370 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.572587 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.572588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.572746 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.572806 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.572808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.573623 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.573624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.574270 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.574273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.574697 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.574698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.574767 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.575090 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.575094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.576855 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.578459 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.578459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.578460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.578611 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.578779 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.578780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.578780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.578846 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.579096 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.579097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.579134 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.579195 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.579301 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:48:48.579455 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.572313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.0.98.61:43707)
0928 22:48:48.572453 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.572458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.572475 (+ 17us) server_negotiation.cc:407] Connection header received
0928 22:48:48.572493 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.572495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.572506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.572564 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.572876 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.573547 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.574339 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.574642 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.574691 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.576936 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.576945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.576945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.576951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.578381 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.578678 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.578682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.578682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.578728 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.578892 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.578893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.578894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.579038 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.579108 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.579237 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.579351 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 22:48:48.579999 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:48.580333 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.580224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56762)
0928 22:48:48.580266 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.580311 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:48:48.580446 23035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
I20260928 22:48:48.580821 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.580713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56772)
0928 22:48:48.580773 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.580796 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:48:48.580914 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37007: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 22:48:48.589457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskntG_DW/build/release/bin/kudu
/tmp/dist-test-taskntG_DW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37007
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:48:48.630134 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } attempt: 1
I20260928 22:48:48.630642 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.630510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56784)
0928 22:48:48.630570 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.630611 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:48:48.630954 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.630872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56788)
0928 22:48:48.630912 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.630934 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:48:48.631039 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37007: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
I20260928 22:48:48.638264 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } attempt: 3
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:48:48.644593 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.638625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.0.98.61:43707)
0928 22:48:48.638685 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.638686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.638698 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:48.638736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.638737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.638749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.638794 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.639091 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.639583 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.640263 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.640475 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.640530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.642327 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.642328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.642329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.642337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.643598 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.643906 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.643907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.643908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.643958 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.644119 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.644120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.644120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.644229 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.644284 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.644420 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.644497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:48.644593 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.638559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45456)
0928 22:48:48.638621 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.638630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.638677 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.638802 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.638803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.638955 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.639019 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.639021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.639667 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.639668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.640188 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.640189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.640537 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.640538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.640608 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.641035 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.641040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.642251 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.643679 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.643680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.643681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.643845 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.644013 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.644013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.644014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.644070 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.644284 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.644286 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.644334 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.644392 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.644496 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:48:48.645148 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:48:48.645480 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.645387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56796)
0928 22:48:48.645432 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.645460 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:48:48.645797 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.645708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56800)
0928 22:48:48.645751 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.645774 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:48:48.645877 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37007: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
W20260928 22:48:48.668442 23094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:48:48.668670 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:48.668696 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:48.668741 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:48.669925 23094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:48.669970 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:48.669988 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:48.670003 23094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:48.670018 23094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:48.671536 23094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34103,127.0.98.61:43707,127.0.98.60:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:48.671789 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:48.671993 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:48.676566 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:48.676604 23094 server_base.cc:1034] running on GCE node
W20260928 22:48:48.676664 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:48.676512 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:48.676983 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:48.677003 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } attempt: 1
I20260928 22:48:48.677271 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:48.677371 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.677257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56806)
0928 22:48:48.677308 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.677339 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 22:48:48.677670 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.677598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56820)
0928 22:48:48.677635 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.677656 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:48:48.677747 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37007: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
I20260928 22:48:48.678416 23094 hybrid_clock.cc:648] HybridClock initialized: now 1790635728678402 us; error 29 us; skew 500 ppm
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:48.680593 23094 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 22:48:48.681010 23094 webserver.cc:460] Webserver started at http://127.0.98.60:34361/ using document root <none> and password file <none>
I20260928 22:48:48.681221 23094 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:48.681267 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:48.681386 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:48.682446 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data/instance:
uuid: "c310926ca6654de9ae38c063a1564275"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "7d63015a786c725cc24503ea018c4a07"
server_key_iv: "a5a16dbd81deaf906538862873ce85d3"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.682760 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal/instance:
uuid: "c310926ca6654de9ae38c063a1564275"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "7d63015a786c725cc24503ea018c4a07"
server_key_iv: "a5a16dbd81deaf906538862873ce85d3"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.684020 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:48.684739 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:48.684890 23094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:48.684966 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal
uuid: "c310926ca6654de9ae38c063a1564275"
format_stamp: "Formatted at 2026-09-28 22:48:48 on dist-test-slave-pxr1"
server_key: "7d63015a786c725cc24503ea018c4a07"
server_key_iv: "a5a16dbd81deaf906538862873ce85d3"
server_key_version: "encryptionkey@0"
I20260928 22:48:48.685026 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:48.699074 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:48.699177 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } attempt: 2
I20260928 22:48:48.699596 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.699460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56828)
0928 22:48:48.699521 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.699561 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:48:48.699895 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.699807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56838)
0928 22:48:48.699848 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.699873 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:48:48.699990 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37007: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37007: connect: Connection refused (error 111)
I20260928 22:48:48.701241 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:48.701368 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:48.704983 23094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37007
I20260928 22:48:48.705013 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37007 every 8 connection(s)
I20260928 22:48:48.705348 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/data/info.pb
I20260928 22:48:48.706284 23162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:48.707772 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:48.713812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskntG_DW/build/release/bin/kudu as pid 23094
I20260928 22:48:48.713905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskntG_DW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635468897675-392-0/minicluster-data/master-2/wal/instance
I20260928 22:48:48.714133 392 external_mini_cluster.cc:1428] Setting key 57492b7052465876e86f29c02ba6602d
I20260928 22:48:48.716305 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.708411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.62:34103)
0928 22:48:48.708474 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.708475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.708667 (+ 192us) server_negotiation.cc:407] Connection header received
0928 22:48:48.709262 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.709264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.709278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.709359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.710017 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.710485 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.711384 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.711557 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.711601 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.713565 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.713567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.713568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.713575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.715089 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.715438 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.715439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.715439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.715500 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.715753 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.715754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.715754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.715854 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.715901 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.716182 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.716242 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:48:48.716825 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.708390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.0.1:34934)
0928 22:48:48.708549 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.708598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.709168 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.709427 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.709431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.709746 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.709939 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.709943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.710547 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.710547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.711325 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.711327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.711618 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.711618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.711722 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.712134 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.712142 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.713478 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.715152 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.715155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.715157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.715371 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.715568 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.715569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.715570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.715693 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.715907 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.715912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.716064 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.716168 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.716575 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":217,"thread_start_us":96,"threads_started":1}
I20260928 22:48:48.717352 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 22:48:48.723027 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.714535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.98.60:37007)
0928 22:48:48.714691 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.714696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.714747 (+ 51us) server_negotiation.cc:407] Connection header received
0928 22:48:48.714812 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.714814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.714825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.714886 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.715520 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.716667 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.717621 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.718007 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.718071 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.720513 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.720523 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.720523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.720530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.721960 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.722296 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.722300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.722300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.722356 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.722514 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.722515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.722515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.722652 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.722784 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.722854 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.722926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260928 22:48:48.723927 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.717592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43707 (local address 127.0.0.1:45470)
0928 22:48:48.717690 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.717698 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.717732 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.717841 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.717842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.717947 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.717994 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.717995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.718588 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.718589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.719363 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.719366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.719730 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.719731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.719784 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.720224 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.720228 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.721463 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.722902 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.722903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.722903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.723080 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.723355 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.723356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.723356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.723411 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.723641 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.723642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.723682 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.723749 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.723837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 22:48:48.723951 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.717648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.98.61:43707)
0928 22:48:48.717712 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.717713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.717725 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:48.717773 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.717774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.717784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.717839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.718051 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.718531 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.719439 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.719659 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.719720 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.721553 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.721554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.721554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.721562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.722827 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.723245 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.723246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.723246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.723296 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.723462 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.723463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.723464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.723584 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.723637 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.723786 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.723868 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:48:48.724303 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 22:48:48.730461 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.724539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56856)
0928 22:48:48.724584 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.724593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.724634 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.724766 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.724766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.724881 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.724905 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.724906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.725710 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.725711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.726335 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.726337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.726658 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.726659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.726700 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.727018 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.727021 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.728036 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.729563 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.729564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.729564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.729710 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.729889 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.729889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.729890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.729933 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.730167 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.730168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.730212 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.730275 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.730361 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:48:48.730530 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.724571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.0.98.60:37007)
0928 22:48:48.724598 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.724600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.724622 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:48:48.724681 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.724682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.724690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.724762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.724963 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.725631 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.726389 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.726603 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.726672 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.728113 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.728115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.728115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.728121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.729483 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.729774 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.729775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.729776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.729829 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.729983 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.729984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.729984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.730100 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.730147 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.730305 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.730413 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":14}
I20260928 22:48:48.731184 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.723750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.98.62:34103)
0928 22:48:48.723795 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.723797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.723842 (+ 45us) server_negotiation.cc:407] Connection header received
0928 22:48:48.723900 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.723901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.723912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.723965 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.724487 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.725060 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.725920 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.726225 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.726292 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.728697 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.728699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.728700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.728707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.729963 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.730240 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.730241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.730242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.730294 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.730469 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.730470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.730470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.730763 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.730988 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.731020 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.731094 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:48.732020 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: Bootstrap starting.
I20260928 22:48:48.732870 23162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:48.733137 23162 log.cc:824] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:48.733824 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: No bootstrap required, opened a new log
I20260928 22:48:48.735566 23162 raft_consensus.cc:348] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.735697 23162 raft_consensus.cc:729] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c310926ca6654de9ae38c063a1564275, State: Initialized, Role: FOLLOWER
I20260928 22:48:48.735796 23162 consensus_queue.cc:260] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.736060 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.736119 23162 sys_catalog.cc:564] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:48.736163 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260928 22:48:48.738402 23179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:48.738456 23179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:48.740130 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.732802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.61:43707)
0928 22:48:48.732850 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.732852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.732859 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:48:48.732880 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.732881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.732889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.732941 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.733413 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.733879 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.734538 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.734870 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.734942 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.737281 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.737283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.737283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.737291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.738943 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.739357 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.739359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.739359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.739424 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.739622 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.739623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.739624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.739766 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.739928 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.739966 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.740043 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:48:48.738272 23094 master_runner.cc:186] Error getting master registration for 127.0.98.62:34103: OK, instance_id { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" instance_seqno: 1790635728411539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:48:48.740471 23094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:48:48.741561 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:48:48.742636 22986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a8dfa536e0334c8ca45a754cd44ed753"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34946
I20260928 22:48:48.752897 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } attempt: 2
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:48:48.760952 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.753345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.98.60:37007)
0928 22:48:48.753389 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.753391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.753402 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:48:48.753478 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.753479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.753489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.753539 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.753873 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.754604 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.755587 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.755890 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.755948 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.758149 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.758151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.758151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.758160 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.759728 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.760113 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.760113 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.760114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.760170 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.760338 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.760338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.760339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.760534 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.760598 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.760769 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.760854 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:48.760957 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.753266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56860)
0928 22:48:48.753332 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.753345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.753409 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.753552 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.753553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.753744 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.753801 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.753803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.754682 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.754683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.755508 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.755510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.755954 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.755955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.756014 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.756509 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.756514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.758033 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.759813 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.759814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.759815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.760036 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.760224 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.760225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.760225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.760285 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.760595 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.760595 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.760644 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.760742 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.760852 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:48:48.762746 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: Bootstrap starting.
I20260928 22:48:48.763514 23012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:48.763816 23012 log.cc:824] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:48.764695 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: No bootstrap required, opened a new log
I20260928 22:48:48.766096 23012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.766258 23012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8dfa536e0334c8ca45a754cd44ed753, State: Initialized, Role: FOLLOWER
I20260928 22:48:48.766351 23012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.766676 23012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:48.766637 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.766754 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:48:48.766794 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:48.766855 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:48.767894 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.768157 22986 leader_election.cc:288] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [CANDIDATE]: Term 1 election: Requested vote from peers 0e18849b79bb4c08a578470f5cc260c5 (127.0.98.61:43707), c310926ca6654de9ae38c063a1564275 (127.0.98.60:37007)
W20260928 22:48:48.768893 23193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 22:48:48.768832 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8dfa536e0334c8ca45a754cd44ed753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c310926ca6654de9ae38c063a1564275"
W20260928 22:48:48.768941 23193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:48.768963 23136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:48.769870 23136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8dfa536e0334c8ca45a754cd44ed753 in term 1.
I20260928 22:48:48.770157 22960 leader_election.cc:302] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8dfa536e0334c8ca45a754cd44ed753, c310926ca6654de9ae38c063a1564275; no voters:
I20260928 22:48:48.770274 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:48.770393 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [term 1 LEADER]: Becoming Leader. State: Replica: a8dfa536e0334c8ca45a754cd44ed753, State: Running, Role: LEADER
I20260928 22:48:48.770470 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.770861 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8dfa536e0334c8ca45a754cd44ed753. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.770927 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:48.771591 22944 master_runner.cc:186] Error getting master registration for 127.0.98.61:43707: OK, instance_id { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" instance_seqno: 1790635728558620 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:48:48.773912 22944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:48:48.776391 23201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:48.776908 23201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:48.782534 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.774320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.98.62:34103)
0928 22:48:48.774357 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.774359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.774971 (+ 612us) server_negotiation.cc:407] Connection header received
0928 22:48:48.775065 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.775066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.775075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.775103 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.775785 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.776489 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.777629 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.777929 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.777972 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.779506 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.779508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.779508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.779516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.781135 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.781597 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.781598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.781598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.781656 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.781901 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.781901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.781902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.782034 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.782105 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.782339 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.782434 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:48.785501 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.771455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.0.98.61:43707)
0928 22:48:48.771501 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.771503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.774300 (+ 2797us) server_negotiation.cc:407] Connection header received
0928 22:48:48.774444 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.774445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.774456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.774498 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.775677 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.776372 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.777951 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.779070 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.779115 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.781680 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.781681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.781682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.781690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.783689 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.784241 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.784242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.784242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.784311 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.784764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.784766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.784766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.784925 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.785012 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.785312 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.785400 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:48:48.790053 23136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [term 1 FOLLOWER]: Refusing update from remote peer a8dfa536e0334c8ca45a754cd44ed753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:48.790288 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.774663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.0.98.60:37007)
0928 22:48:48.774701 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.774716 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.778329 (+ 3613us) server_negotiation.cc:407] Connection header received
0928 22:48:48.778422 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.778424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.778434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.778462 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.783637 (+ 5175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.784387 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.785448 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.785756 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.785796 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.787306 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.787307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.787308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.787315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.788991 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.789407 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.789408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.789408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.789487 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.789681 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.789682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.789682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.789810 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.789900 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.790108 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.790190 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:48:48.790809 23182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:48.791122 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } attempt: 3
I20260928 22:48:48.793630 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8dfa536e0334c8ca45a754cd44ed753. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.793742 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 22:48:48 dist-test-slave-pxr1 krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 22:48:48.799717 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.799834 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:48.799902 23205 mvcc.cc:205] Tried to move back new op lower bound from 7334443945120157696 to 7334443945044582400. Current Snapshot: MvccSnapshot[applied={T|T < 7334443945120157696}]
I20260928 22:48:48.800920 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.801009 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:48.801259 23206 mvcc.cc:205] Tried to move back new op lower bound from 7334443945120157696 to 7334443945044582400. Current Snapshot: MvccSnapshot[applied={T|T < 7334443945120157696}]
I20260928 22:48:48.802966 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.791373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37007 (local address 127.0.0.1:56878)
0928 22:48:48.791423 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:48.791433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:48.791476 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:48.791716 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:48.791717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:48.791896 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:48.791944 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.791946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.794484 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.794485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.795327 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:48.795330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.795838 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.795839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:48.795900 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.796377 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:48.796382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:48.797888 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:48.801637 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.801638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.801639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.801878 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.802148 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:48.802149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:48.802149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:48.802209 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:48.802626 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:48.802627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:48.802698 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 22:48:48.802754 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:48:48.802866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:48:48.803591 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:48.791399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.60:37007)
0928 22:48:48.791430 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.791432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.791462 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:48:48.791582 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.791583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.791592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.791647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.792061 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.793257 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.795424 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:48.795722 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:48.795780 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:48.797987 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:48.797989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:48.797989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:48.797997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:48.801499 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.801961 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.801963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.801963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.802043 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:48.802279 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:48.802280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:48.802280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:48.802500 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:48.803150 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:48.803218 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:48:48.803344 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 22:48:48.805433 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: Bootstrap starting.
I20260928 22:48:48.805719 23201 catalog_manager.cc:1347] Generated new cluster ID: 4bfedc81330a4eb8b8c60917d99043fe
I20260928 22:48:48.805783 23201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:48.806174 23087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:48.806476 23087 log.cc:824] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:48.809499 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: No bootstrap required, opened a new log
I20260928 22:48:48.811537 23087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.811725 23087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e18849b79bb4c08a578470f5cc260c5, State: Initialized, Role: FOLLOWER
I20260928 22:48:48.811816 23087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } }
I20260928 22:48:48.812162 23087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:48.812278 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:48.812783 23061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8dfa536e0334c8ca45a754cd44ed753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e18849b79bb4c08a578470f5cc260c5"
I20260928 22:48:48.813189 23060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [term 1 FOLLOWER]: Refusing update from remote peer a8dfa536e0334c8ca45a754cd44ed753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:48.813588 23195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:48.814096 23061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8dfa536e0334c8ca45a754cd44ed753 in term 1.
I20260928 22:48:48.812158 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.814248 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:48.816066 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8dfa536e0334c8ca45a754cd44ed753. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.816147 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:48.816224 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.816319 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:48:48.816910 23226 catalog_manager.cc:1259] Loaded cluster ID: 4bfedc81330a4eb8b8c60917d99043fe
I20260928 22:48:48.816949 23226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: loading cluster ID for follower catalog manager: success
W20260928 22:48:48.817019 23226 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 22:48:48.817049 23226 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:48.826377 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8dfa536e0334c8ca45a754cd44ed753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8dfa536e0334c8ca45a754cd44ed753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } peers { permanent_uuid: "0e18849b79bb4c08a578470f5cc260c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43707 } } peers { permanent_uuid: "c310926ca6654de9ae38c063a1564275" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37007 } } }
I20260928 22:48:48.826686 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:48.829497 23201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:48.830394 23201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:48.837641 23201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8dfa536e0334c8ca45a754cd44ed753: Generated new TSK 0
I20260928 22:48:48.837880 23201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:48:48.862143 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.860226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.0.98.60:37007)
0928 22:48:48.860276 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.860278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.861428 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 22:48:48.861535 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.861537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.861548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.861579 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.862067 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56882: BlockingRecv error: recv got EOF from 127.0.0.1:56882 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 22:48:48.862957 23015 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.860500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.0.98.62:34103)
0928 22:48:48.860542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.860544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.862417 (+ 1873us) server_negotiation.cc:407] Connection header received
0928 22:48:48.862511 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.862512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.862523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.862550 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.862890 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34958: BlockingRecv error: recv got EOF from 127.0.0.1:34958 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:48:48.863744 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.860791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.0.98.61:43707)
0928 22:48:48.860837 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.860839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.863195 (+ 2356us) server_negotiation.cc:407] Connection header received
0928 22:48:48.863309 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.863311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.863322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.863350 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.863672 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45486: BlockingRecv error: recv got EOF from 127.0.0.1:45486 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 22:48:48.866357 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.865494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.0.98.61:43707)
0928 22:48:48.865533 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.865536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.865788 (+ 252us) server_negotiation.cc:407] Connection header received
0928 22:48:48.865894 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.865895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.865914 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.865945 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.866305 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45496: BlockingRecv error: recv got EOF from 127.0.0.1:45496 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:48:48.870857 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.869934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.98.61:43707)
0928 22:48:48.869976 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.869977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.870271 (+ 294us) server_negotiation.cc:407] Connection header received
0928 22:48:48.870375 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.870376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.870388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.870417 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.870796 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45502: BlockingRecv error: recv got EOF from 127.0.0.1:45502 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:48:48.875154 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.874267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.0.98.61:43707)
0928 22:48:48.874307 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.874308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.874581 (+ 273us) server_negotiation.cc:407] Connection header received
0928 22:48:48.874702 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.874703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.874716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.874745 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.875099 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45518: BlockingRecv error: recv got EOF from 127.0.0.1:45518 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:48:48.881393 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.880425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.0.98.61:43707)
0928 22:48:48.880468 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.880469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.880788 (+ 319us) server_negotiation.cc:407] Connection header received
0928 22:48:48.880902 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.880903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.880917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.880947 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.881333 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45534: BlockingRecv error: recv got EOF from 127.0.0.1:45534 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:48:48.891681 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.890724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.0.98.61:43707)
0928 22:48:48.890770 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.890772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.891077 (+ 305us) server_negotiation.cc:407] Connection header received
0928 22:48:48.891184 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.891185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.891197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.891226 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.891618 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45544: BlockingRecv error: recv got EOF from 127.0.0.1:45544 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:48:48.919275 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.914129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.0.98.61:43707)
0928 22:48:48.914177 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.914179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.918591 (+ 4412us) server_negotiation.cc:407] Connection header received
0928 22:48:48.918720 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.918721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.918735 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.918768 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.919206 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45556: BlockingRecv error: recv got EOF from 127.0.0.1:45556 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:48:48.953285 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:48.952561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.0.98.61:43707)
0928 22:48:48.952606 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:48.952608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:48.952738 (+ 130us) server_negotiation.cc:407] Connection header received
0928 22:48:48.952836 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:48.952837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:48.952848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:48.952892 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:48.953222 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45572: BlockingRecv error: recv got EOF from 127.0.0.1:45572 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:48:49.018976 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:49.018379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.0.98.61:43707)
0928 22:48:49.018450 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.018451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.018516 (+ 65us) server_negotiation.cc:407] Connection header received
0928 22:48:49.018571 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.018572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.018583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.018649 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:49.018927 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45576: BlockingRecv error: recv got EOF from 127.0.0.1:45576 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 22:48:49.148870 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:49.148107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.98.61:43707)
0928 22:48:49.148163 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.148165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.148299 (+ 134us) server_negotiation.cc:407] Connection header received
0928 22:48:49.148333 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.148334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.148345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.148419 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:49.148807 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45580: BlockingRecv error: recv got EOF from 127.0.0.1:45580 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 22:48:49.406854 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:48:49.406202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.0.98.61:43707)
0928 22:48:49.406270 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.406272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.406286 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:48:49.406347 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.406348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.406358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.406401 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:49.406788 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45582: BlockingRecv error: recv got EOF from 127.0.0.1:45582 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:48:49.738891 23179 catalog_manager.cc:1259] Loaded cluster ID: 4bfedc81330a4eb8b8c60917d99043fe
I20260928 22:48:49.738974 23179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: loading cluster ID for follower catalog manager: success
I20260928 22:48:49.739755 23179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: acquiring CA information for follower catalog manager: success
I20260928 22:48:49.740026 23179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c310926ca6654de9ae38c063a1564275: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:48:49.818097 23226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: acquiring CA information for follower catalog manager: success
I20260928 22:48:49.818454 23226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e18849b79bb4c08a578470f5cc260c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:48:49.922647 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:49.919993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.0.98.61:43707)
0928 22:48:49.920115 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.920117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.920128 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:48:49.920198 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.920199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.920247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.920319 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:49.920479 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.921047 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.921854 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.922062 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.922124 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:49.922292 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:49.922404 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:49.922459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:49.922531 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:48:49.922560 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260928 22:48:49.927575 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:49.924260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.98.60:37007)
0928 22:48:49.924356 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.924358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.924411 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:48:49.924484 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.924484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.924535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.924614 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:49.924782 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.925282 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.926460 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.926782 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.926859 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:49.927068 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:49.927219 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:49.927302 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:49.927343 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:48:49.927368 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260928 22:48:49.933630 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:49.930393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.61:43707)
0928 22:48:49.930443 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.930445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.930621 (+ 176us) server_negotiation.cc:407] Connection header received
0928 22:48:49.930811 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.930813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.930827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.930856 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:49.931129 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.931782 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.932849 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.933153 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.933194 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:49.933388 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:49.933420 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:49.933443 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:49.933527 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 22:48:49.933550 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:48:49.933683 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:49.930222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.98.60:37007)
0928 22:48:49.930385 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.930387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.930600 (+ 213us) server_negotiation.cc:407] Connection header received
0928 22:48:49.930730 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.930731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.930742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.930770 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:49.930942 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.931705 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.932872 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.933158 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.933202 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:49.933408 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:49.933456 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:49.933481 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:49.933580 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:48:49.933606 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 22:48:49.934795 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:49.930187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.62:34103)
0928 22:48:49.931556 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:49.931558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:49.931567 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:49.931624 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:49.931625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:49.931674 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:49.931754 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:49.931925 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.932680 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.933681 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:49.934249 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:49.934310 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:49.934489 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:49.934587 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:49.934635 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:49.934686 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 22:48:49.934709 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":52,"threads_started":1}
I20260928 22:48:49.936492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 22944
I20260928 22:48:49.941325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 23019
I20260928 22:48:49.945940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskntG_DW/build/release/bin/kudu with pid 23094
2026-09-28T22:48:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2740 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5457 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (261052 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST